Sc3ne

So, you want to scrape (save) everything from Google Plus, huh? True, if you don’t have much content you won’t need to do this. In that case, you can just ctrl-a to select it all, then control-c to copy it and control-v to paste it into Microsoft Word… however, even then you’ll lose all of the images when copied.

So, what do I need to do it? Adobe Acrobat if you want it in PDF. Plus Firefox, iMacros and a reasonably fast (8 gigs of RAM and an i5 / similar or better) computer. If you want it converted into DOC format, Corel PDF Fusion as well. Yeah, you can just copy and paste into a text editor and then parse the text with whatever technique you want to if you just want the text for data mining or re-use purposes. However, this is aimed more toward the person who has to close their social network account (Google Plus) before applying for a new job or school or someone who just wants it all saved for record-keeping for sentimental value, scrap-booking or hell, even someone who wants to copy someone’s entire Google Plus stream for a court case like in a divorce court.

First, open Firefox. Install iMacros from their Mozilla addon’s repository or go to the iOpus page: www.iopus.com/imacros and follow the instructions there to install it. Once it’s installed and Firefox has been restarted, using the iOpus instructions, create a new macro, open the editor and paste this first code in. Open your Google Plus page or someone else’s, then run it. It’ll open every “page” by clicking the “more” button. Set the macro to loop at 999 times… it generally won’t take that many, but it might if you have 4,000 paper-pages worth of content to grab. 800 “paper-pages” will take roughly 10 to 15 minutes to completely load if you have a reasonably fast PC and Internet connection (I have a 30 megabit cable modem connection), more if not. Everything after the first <–> and before the second <–>.

<–>

 

SET !ERRORIGNORE YES

TAG POS=1 TYPE=SPAN ATTR=CLASS:np

WAIT SECONDS=1

 

<–>

 

So, that code above will open all of the pages. The code below opens up all the posts that have more than one or two comments. Again, after the first <–> and before the second <–>.

 

<–>

 

SET !ERRORIGNORE YES

TAG POS=1 TYPE=SPAN ATTR=CLASS:a-o<SP>Ip<SP>Yz

WAIT SECONDS=1

 

<–>

 

So, the code above opens up all of the comments. The code below will be the last touch, expanding all of the comments.

 

<–>

 

SET !ERRORIGNORE YES

TAG POS=1 TYPE=SPAN ATTR=TABINDEX:0&&CLASS:a-o<SP>hn<SP>eg&&ROLE:button&&TXT:Expand<SP>this<SP>comment<SP>»

WAIT SECONDS=1

 

<–>

now, this code above is flawed because of the limitation of iMacros. essentially, they have no easy solution to batch jobs that require dynamic content in the macro itself without resorting to other programming languages, which is a problem with EVERY other platform out there… incorporating Regular Expressions for any part of the code would fix it, but that’s not what we’re talking about here. Back on topic, because that’s a limitation, the code has to be longer.

 

you’ll need a list of sequential numbers, as many as you conceive you might have to use for the number of comments you’ll have to expand. On 800 pages worth of content, that was around 400 in my case.

 

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19

 

That’s what I mean by list of sequential numbers. Use this generator here at TextMechanic: textmechanic.com/Generate-List-of-Numbers.html , it’s easy and intuitive to use.

Then, use a RegEx text editor (my favorite is EditPad Pro) to add each parts of the code above around the numbers (RegEx replace ^ for before the start of the line, $ for at the end of the line). In iMacros, POS=<number here> tells the macro which instance of what it’s searching for to target. So, POS=1 is the first, POS=2 is the second, POS=3 is the third and so on. It’s a hacky solution, but the only way to go about it with iMacros. Which for any large batch job with iMacros is a severe problem because the editor seems to hate large files. Hell, even after editing the macro in EditPad and saving it with the .IIM extension for iMacros, iMacros prompted me to be sure I wanted to load such a “big” file at all of 2,000 lines of a macro…

 

So it’ll look something like this, but the POS will be numbered 1 through whatever you may need (200 or 1,000? possibly):

 

<–>

 

SET !ERRORIGNORE YES

TAG POS=45 TYPE=SPAN ATTR=TABINDEX:0&&CLASS:a-o<SP>hn<SP>eg&&ROLE:button&&TXT:Expand<SP>this<SP>comment<SP>»
WAIT SECONDS=1
TAG POS=46 TYPE=SPAN ATTR=TABINDEX:0&&CLASS:a-o<SP>hn<SP>eg&&ROLE:button&&TXT:Expand<SP>this<SP>comment<SP>»
WAIT SECONDS=1
TAG POS=47 TYPE=SPAN ATTR=TABINDEX:0&&CLASS:a-o<SP>hn<SP>eg&&ROLE:button&&TXT:Expand<SP>this<SP>comment<SP>»
WAIT SECONDS=1
TAG POS=48 TYPE=SPAN ATTR=TABINDEX:0&&CLASS:a-o<SP>hn<SP>eg&&ROLE:button&&TXT:Expand<SP>this<SP>comment<SP>»

(and so on…)

 

<–>

 

So, after all that’s done, “print” the huge ass webpage that you used iMacros to open up and expand all of the JQuery-run elements on. Use the Adobe Acrobat printing tool. If you want, after that you can open that PDF in Corel PDF Fusion and then save it as a DOC file. Or if you just want the next and not the images or people’s avatar icons next to their posts, you can just copy and paste it all into MS Word or a Text Editor. A note: If you have a lot of pages, Microsoft One Note won’t hack it, but I had limited success copying all of the images and text together, keeping the formatting for the most part as well. From One Note I was able to save as a DOCX file or export as a PDF. However, for accounts with even 1 post every few days and a fair amount (even 15) of commenters on many posts, it would be a poor solution, especially to save EVERYTHING and with the formatting you get when you look at your Google Plus page and post history. And while yes, I could’ve had iMacros save the text entries into a CSV file to use as a spreadsheet or into an organized text file, I wanted to save the entire set of web content pages so that they had the same impact as when originally posted and interacted with, effectively backing up some of the Google Plus experience as well. OCR can in the future pull out some of the text or if you want to save the text separately, ctrl-a and ctrl-c into a word document or text file. Keep in mind word documents are bigger than the PDFs will be. PDFs will be 65ish megabytes for even 800 pages of content (depending on compression and resolution of the capture, that’s what it was after output from Acrobat’s printing to file software) whereas word DOCs created from Corel PDF Fusion around 800 pages are 180-ish megabytes.

The End.

 

Also, here are my thoughts on the best of web scraping and automation platforms:

www.sc3ne.com/useful-software/web-automation-and-scraping-tools-my-top-7/