Exporting Bookmarks STRUCTURE, AS WELL AS content

[DickB36]

I need to be able to export (extract) the following info from FF's "Bookmarks" utility:

1. Name
2. Location (url)
3. Links between Bookmarks and the "nodes" to which they are connected

Using the "Export" option creates an .HTML file that DISPLAYS only the Names (as hyper-text links) -- but not ALSO the locations; OR which Bookmarks are connected to which notes in the Bookmarks "tree."

Is there any Add-on, or other utility, that can do this?

Any suggestions?

I need to be able to get to this info - in some recognizably-delimited format, so I can use it in an app I'm working on which will, among other things, resolve all of the "management-and-retrieval" problems with the FF Bookmarks utility.

DickB36

[jscher2000]

I can't think of a built-in feature for this. There may well be an add-on that can do it.

Otherwise, could you clarify what you mean by #3? Is it the name of the folder that contains the bookmark, or something more than that?

[JonD]

I have pondered the same question in the past. The page http://justsolve.archiveteam.org/wiki/Firefox_bookmarks describes how and where bookmarks are stored, so presumably if you issue SQL commands into the SQLite database, you can achieve what the OP is seeking.  I haven't tried this though.  Hope that helps,

Jon

[DickB36]

JonC

Thank you!  I will check this out.

I'll let you know what (if anything) I  find.

sdickB34

[DickB36]

JonB,

Thank you for  the tips about the names and location  of the Firefox “Bookmarks” file (e.g., bookmarks-2012-10-28.jsonlz4, places.sqlite and places.sqlite-wal) —  and for letting me know that the Ff “Bookmarks” library’s “Backup” option produces an UNENCRYPTED version of the   .jsonlz4 file: bookmarks-2012-10-28.json.    At least you van SEE what you’re dealing with!

Ff Bookmarks are stored in a hierarchical “tree” structure, which can have nodes, sub-nodes, nodes-of-sub-nodes…  In this setting, the “context” of any “bookmark”  — its “meaning” — is determined by its POSITION in the tree.  Copies of any bookmark can be  stored in as many nodes as seems suitable.

Ff nodes define SETS of bookmarks.

In  a .JSON (comma-delimited) file, “records” are defined this way:

   Every record begins with the notation: {guid":

   Every record ends with the notation: “}"

   With this exception: a record can also end with the notation: "children":[

   “[“ indicates the beginning of a SET

   ]“ indicates the end of a SET

The rest of the “translation” work involves “parsing each JSON record, and stripping-out the data you need.  More on this, later (I hope)

Ff Bookmarks are stored in a hierarchical “tree” structure, which can have nodes, sub-nodes, nodes-of-sub-nodes…  In this setting, the “context” of any “bookmark”  — its “meaning” — is determined by its POSITION in the tree.  Copies of any bookmark can be  stored in as many nodes as seems suitable.

Ff nodes define SETS of bookmarks

I’m trying to store this same information as records in a data-file.  SETS of bookmarks are defined by LINKING any bookmark to one or more records in another data-table (e.g., “record-A”).  In this way, record-A (and other records in the same data-file) defines the SET of bookmarks LINKED to that record.

[jscher2000]

---

@DickB36 wrote:

I’m trying to store this same information as records in a data-file.  SETS of bookmarks are defined by LINKING any bookmark to one or more records in another data-table (e.g., “record-A”).  In this way, record-A (and other records in the same data-file) defines the SET of bookmarks LINKED to that record.

---

What is your desired data structure? I understand the first two, but what is the third one?

• Title
• URL
• node [is this a folder name?]

In a relational database, you generally would use a separate table to store relationships among the bookmarks/folders. Firefox does that in the Places database (places.sqlite) that store bookmarks along with history, but when you use Export or Backup, the original tables are not replicated in the file.

I have a page that reads the JSON-format backup file and allows exporting a file in the HTML format. I just added a new feature to export selected columns to a tab-delimited ("TSV") file. This can be read in to Excel using the Data tab, or presumably you can work with it in other ways. Maybe worth a try?

https://www.jeffersonscher.com/ffu/bookbackreader.html

[DickB36]

Ischer2000,

Thanks. To answer your question, first: a "node" is a FOLDER, a "branch,"  in the Bookmarks Library "tree"; it's where any given bookmark is stored (any bookmark may be stored in many nodes).  Any node can also contain sub-nodes; sub-nodes... (as well as store its own "leaves": the bookmarks).

In JSON-speak, these nodes are designated by the notation "children":[.................]

Re: bookbackreader.html

That (especially the conversion to a .TSV file) sounds like interesting work!

For what I need, converting a .JSON file to ,HTML (and maybe even to a TSV) would still leave me with the problem of having to "peel" the individual data-file "records" (now stored in the .HTML) -- which is what I'm after.

I've just-about finished coding the program which will perform the first step of this process. 

It breaks-out JSON "records" (the JSON file is ONE CONTINUOUS, multi-megabyte, STRING that contains individually-delimited "records") and stores these in an intermediary data-table; where I can actually SEE the information I have.

Then I have to figure-out the next step: "translating" this info and storing IT into  next intermediary table.  And, ultimately, into the final data-file that will store "bookmarks."  I already have the other files you mentioned -- data-fie, linked-list (m:m) file -- that I need to complete the translation ("re-interpretation") of the Firefox "Bookmarks" utility.

This will also do a lot more than that.

I expect the entire conversion-operation will take four such steps.  I THINK this first step will be the toughest; it involves dealing with the conventions of a unfamiliar "language": the JSON file structure.

[jscher2000]

---

@DickB36 wrote:

Re: bookbackreader.html

That (especially the conversion to a .TSV file) sounds like interesting work!

For what I need, converting a .JSON file to ,HTML (and maybe even to a TSV) would still leave me with the problem of having to "peel" the individual data-file "records" (now stored in the .HTML) -- which is what I'm after.

---

Right, HTML is not a a good endpoint for you. TSV is similar to CSV but instead of delimiters being commas, they are tabs.

It's open source, so you can see/borrow the recursive logic. In case it's easier:

https://github.com/jscher2000/Firefox-File-Utilities/blob/master/bookbackreader.html#L396

[DickB36]

Ischer2000,

What do I need to be able to run your code? I read id; it looks nicely-written (I've been coding for 40 years) -- but it's not a language I use.

The only JSON info I need to extract is: Title, URL, Folder Path. 

I'm not trying to reproduce the Ff Bookmarks HIERARCHY; only the Bookmark Titles, URLs, and how  they're organized .

TAB-delimited makes it easier to SEE the individual "fields" in JSON file delimited "records."  This is what the code, which I think I can get ready to test, tonight, will do.  But, if you have something that already works....!

Remember, I need to ultimately get this info into the data-file tables of at least a couple of the apps I'm developing.  Where does your  stuff leave-off, relative to this?

[jscher2000]

---

@DickB36 wrote:

What do I need to be able to run your code?

---

It's JavaScript, designed to run in a browser. You could just try out the page as an end user: in Step 1, drop your JSON or JSONLZ4 file onto the box, then scroll down to the Tab-separated list section, check the Folder Path box, and click the Generate button Firefox should display a download dialog to save the file and you can see how the Folder Path is displayed.

[DickB36]

Ischer32000,What especially interests e about your utility is that it captures the FOLDER PATH.

Fol der_A

      bookmark

      bookmark

      Folder_B

         bookmark

          bookmark

          Folder_C

           bookmark

           bookmark

            Bookmark_A

            …………..

            bookmark

    Folder_D

 All I need to know is how the information generated by your program visually represents this path.   I can see what JSON-file info you use to do identify the FP; but I don’t know how you do this. 

But, I have other ways to ”recreate” this containership info (and to support ordering-within-folders); and I don’t need to recreate the Firefox Bookmarks hierarchy. 

I’m just trying to get clear on how your utility can help me.

[DickB36]

I meant to mention that I need to identify the FOLDER PATH to Bookmark_A (and any other Bookmarks).

[DickB36]

"Drop your JSON or JSONLZ4 file onto the box"???

When I click on the link you sent me, I get to a page with your source code (on GitHub).

When I select "Got o File," II get to a GitHub) Firefox Utilities page.

WHICH utility do I select?  Where can I enter the fullpath to the .JSON file I want to test?

[jscher2000]

---

@DickB36 wrote:

"Drop your JSON or JSONLZ4 file onto the box"???

When I click on the link you sent me, I get to a page with your source code (on GitHub).

---

You need to use the link I provided originally:

https://www.jeffersonscher.com/ffu/bookbackreader.html

or the Github-hosted version of the page:

https://jscher2000.github.io/Firefox-File-Utilities/bookbackreader.html

[DickB36]

Lovely!

I can get to where I need to much more easily from your .TSV

Thanks!

[DickB36 · Accepted solution]

The Firefox “Manage Bookmarks” utility (Library) has two “sections.”

   1)  The “Tree,” which shows our Folders (branches), and the bookmarks stored in each    Folder

   2) A composite listing of all our bookmarks (at the bottom of “Bookmarks Menu”

I only wanted bookmarks Title, URL and Folder info, so I only “peeled” the records from the “Tree” section.

The utility that Ischer2000 suggested produced  TAB-limited (.TSV) file that extracted this information from a “Backup” copy of the Library (.json), which made it reasonably-easy to capture what I want.

https://www.jeffersonscher.com/ffu/bookbackreader.html

I ran into one problem in the process of converting the Firefox records to a data file:

In the TITLE field, non-priming (and “invisible” characters cropped up in some of these records (and the FOLDER PATH field for all records).    These were easily trapped, and stripped-out. as part of the operation of transferring the .TSV info to a data-file; from where I can perform all of the additional conversion operatios I'll need.

Thank you, Ischer2000!

[jscher2000]

In the Folder path, was the problem the use of right-angle quotation marks to indicate the nesting? You could revise the script to insert a different character.

https://github.com/jscher2000/Firefox-File-Utilities/blob/master/bookbackreader.html#L490

To download the files to use locally, click the green "<> Code" button here and grab the zip file:

https://github.com/jscher2000/Firefox-File-Utilities

[DickB36]

Ischer,

No, the high-ASCII chars. did not include the right-angle marks (I knew that they were parto of what in JSON formatting is referred to as "children":[ (and as TypeCode:2) -- i.e., "folders" (or "nodes").

In the FOLDER PATH segment of the .TSVA string, "Bookmark Menu became "BookMarkA" then a space;  then the angle-brackets.

In the TITLE segment, 2 or 3 non-printable chars. SOMETIMES appear at the beginning of the (TAB-delimited) string) -- and different ASCII chars further-on in the string.

This behavior was NOT consistent.  But, instead of looking for a pattern in its occurrences, I just wrote a function that parsed the entire TITLE string and stripped-out the problem chars.

Otherwise, your utility saved me DAYS of work (thank you for that).  Now I'm working on cleaning-up the occasional damaged record -- which SEEMS to be limited to te END of the FOLDER PATH string:

   "Speeches;S"  "Speeches;Sta"  "Speeches;Stat"  " Speeches;Statements"

These turned-up in 4 CONSECUTIVE bookmarks.  The only possible explanation is that the over-all LENGTH of the block of chars. I retrieved from your file was LESS than the length of the over-all "reecord (upto the CR).  I'll check; but it doesn't seem to be. that.

This corruption of the data seems to be more-extensive (I know that all of these records -- bookmrks -- are in the SAME folder).

Anythoughts?

[DickB36]

Ischer,

All "corrupted" have been fixed. by making 2 corrections:

   1) Increasedthe number of bytes read from your .TSV file (opened as a low-level file: i.e., not by aby app with a User-interface)

   2) Increased the sizes of the data-file fields used to capture the TITLE, URL, and FOLDER PATH values "peeled" from the .TSV file.

Now, all of the info extracted from Firefox appears to be "good"!