topic Re: Exporting Bookmarks STRUCTURE, AS WELL AS content in Discussions

Exporting Bookmarks STRUCTURE, AS WELL AS content

DickB36 — Wed, 12 Apr 2023 09:20:42 GMT

I need to be able to export (extract) the following info from FF's "Bookmarks" utility:

Name
Location (url)
Links between Bookmarks and the "nodes" to which they are connected

Using the "Export" option creates an .HTML file that DISPLAYS only the Names (as hyper-text links) -- but not ALSO the locations; OR which Bookmarks are connected to which notes in the Bookmarks "tree."

Is there any Add-on, or other utility, that can do this?

Any suggestions?

I need to be able to get to this info - in some recognizably-delimited format, so I can use it in an app I'm working on which will, among other things, resolve all of the "management-and-retrieval" problems with the FF Bookmarks utility.

DickB36

Re: Exporting Bookmarks STRUCTURE, AS WELL AS content

jscher2000 — Wed, 12 Apr 2023 16:14:50 GMT

I can't think of a built-in feature for this. There may well be an add-on that can do it.

Otherwise, could you clarify what you mean by #3? Is it the name of the folder that contains the bookmark, or something more than that?

Re: Exporting Bookmarks STRUCTURE, AS WELL AS content

JonD — Wed, 12 Apr 2023 17:35:06 GMT

I have pondered the same question in the past. The page http://justsolve.archiveteam.org/wiki/Firefox_bookmarks describes how and where bookmarks are stored, so presumably if you issue SQL commands into the SQLite database, you can achieve what the OP is seeking. I haven't tried this though. Hope that helps,

Jon

Re: Exporting Bookmarks STRUCTURE, AS WELL AS content

DickB36 — Wed, 12 Apr 2023 21:10:04 GMT

JonC

Thank you! I will check this out.

I'll let you know what (if anything) I find.

sdickB34

Re: Exporting Bookmarks STRUCTURE, AS WELL AS content

DickB36 — Sat, 15 Apr 2023 22:14:52 GMT

JonB,

Thank you for the tips about the names and location of the Firefox “Bookmarks” file (e.g., bookmarks-2012-10-28.jsonlz4, places.sqlite and places.sqlite-wal) — and for letting me know that the Ff “Bookmarks” library’s “Backup” option produces an UNENCRYPTED version of the .jsonlz4 file: bookmarks-2012-10-28.json. At least you van SEE what you’re dealing with!

Ff Bookmarks are stored in a hierarchical “tree” structure, which can have nodes, sub-nodes, nodes-of-sub-nodes… In this setting, the “context” of any “bookmark” — its “meaning” — is determined by its POSITION in the tree. Copies of any bookmark can be stored in as many nodes as seems suitable.

Ff nodes define SETS of bookmarks.

In a .JSON (comma-delimited) file, “records” are defined this way:

Every record begins with the notation: {guid":

Every record ends with the notation: “}"

With this exception: a record can also end with the notation: "children":[

“[“ indicates the beginning of a SET

]“ indicates the end of a SET

The rest of the “translation” work involves “parsing each JSON record, and stripping-out the data you need. More on this, later (I hope)

Ff nodes define SETS of bookmarks

I’m trying to store this same information as records in a data-file. SETS of bookmarks are defined by LINKING any bookmark to one or more records in another data-table (e.g., “record-A”). In this way, record-A (and other records in the same data-file) defines the SET of bookmarks LINKED to that record.

Re: Exporting Bookmarks STRUCTURE, AS WELL AS content

jscher2000 — Mon, 17 Apr 2023 01:08:15 GMT

@DickB36 wrote:
I’m trying to store this same information as records in a data-file. SETS of bookmarks are defined by LINKING any bookmark to one or more records in another data-table (e.g., “record-A”). In this way, record-A (and other records in the same data-file) defines the SET of bookmarks LINKED to that record.

What is your desired data structure? I understand the first two, but what is the third one?

Title
URL
node [is this a folder name?]

In a relational database, you generally would use a separate table to store relationships among the bookmarks/folders. Firefox does that in the Places database (places.sqlite) that store bookmarks along with history, but when you use Export or Backup, the original tables are not replicated in the file.

I have a page that reads the JSON-format backup file and allows exporting a file in the HTML format. I just added a new feature to export selected columns to a tab-delimited ("TSV") file. This can be read in to Excel using the Data tab, or presumably you can work with it in other ways. Maybe worth a try?

https://www.jeffersonscher.com/ffu/bookbackreader.html

Re: Exporting Bookmarks STRUCTURE, AS WELL AS content

DickB36 — Mon, 17 Apr 2023 04:13:14 GMT

Ischer2000,

Thanks. To answer your question, first: a "node" is a FOLDER, a "branch," in the Bookmarks Library "tree"; it's where any given bookmark is stored (any bookmark may be stored in many nodes). Any node can also contain sub-nodes; sub-nodes... (as well as store its own "leaves": the bookmarks).

In JSON-speak, these nodes are designated by the notation "children":[.................]

Re: bookbackreader.html

That (especially the conversion to a .TSV file) sounds like interesting work!

For what I need, converting a .JSON file to ,HTML (and maybe even to a TSV) would still leave me with the problem of having to "peel" the individual data-file "records" (now stored in the .HTML) -- which is what I'm after.

I've just-about finished coding the program which will perform the first step of this process.

It breaks-out JSON "records" (the JSON file is ONE CONTINUOUS, multi-megabyte, STRING that contains individually-delimited "records") and stores these in an intermediary data-table; where I can actually SEE the information I have.

Then I have to figure-out the next step: "translating" this info and storing IT into next intermediary table. And, ultimately, into the final data-file that will store "bookmarks." I already have the other files you mentioned -- data-fie, linked-list (m:m) file -- that I need to complete the translation ("re-interpretation") of the Firefox "Bookmarks" utility.

This will also do a lot more than that.

I expect the entire conversion-operation will take four such steps. I THINK this first step will be the toughest; it involves dealing with the conventions of a unfamiliar "language": the JSON file structure.

Re: Exporting Bookmarks STRUCTURE, AS WELL AS content

jscher2000 — Mon, 17 Apr 2023 16:15:19 GMT

@DickB36 wrote:
Re: bookbackreader.html
That (especially the conversion to a .TSV file) sounds like interesting work!
For what I need, converting a .JSON file to ,HTML (and maybe even to a TSV) would still leave me with the problem of having to "peel" the individual data-file "records" (now stored in the .HTML) -- which is what I'm after.

Right, HTML is not a a good endpoint for you. TSV is similar to CSV but instead of delimiters being commas, they are tabs.

It's open source, so you can see/borrow the recursive logic. In case it's easier:

https://github.com/jscher2000/Firefox-File-Utilities/blob/master/bookbackreader.html#L396

Re: Exporting Bookmarks STRUCTURE, AS WELL AS content

DickB36 — Tue, 18 Apr 2023 02:30:08 GMT

Ischer2000,

What do I need to be able to run your code? I read id; it looks nicely-written (I've been coding for 40 years) -- but it's not a language I use.

The only JSON info I need to extract is: Title, URL, Folder Path.

I'm not trying to reproduce the Ff Bookmarks HIERARCHY; only the Bookmark Titles, URLs, and how they're organized .

TAB-delimited makes it easier to SEE the individual "fields" in JSON file delimited "records." This is what the code, which I think I can get ready to test, tonight, will do. But, if you have something that already works....!

Remember, I need to ultimately get this info into the data-file tables of at least a couple of the apps I'm developing. Where does your stuff leave-off, relative to this?

Re: Exporting Bookmarks STRUCTURE, AS WELL AS content

jscher2000 — Tue, 18 Apr 2023 03:19:51 GMT

@DickB36 wrote:
What do I need to be able to run your code?

It's JavaScript, designed to run in a browser. You could just try out the page as an end user: in Step 1, drop your JSON or JSONLZ4 file onto the box, then scroll down to the Tab-separated list section, check the Folder Path box, and click the Generate button Firefox should display a download dialog to save the file and you can see how the Folder Path is displayed.

Re: Exporting Bookmarks STRUCTURE, AS WELL AS content

DickB36 — Tue, 18 Apr 2023 13:35:52 GMT

Ischer32000,What especially interests e about your utility is that it captures the FOLDER PATH.

Fol der_A

bookmark

Folder_B

bookmark

Folder_C

bookmark

Bookmark_A

…………..

bookmark

Folder_D

All I need to know is how the information generated by your program visually represents this path. I can see what JSON-file info you use to do identify the FP; but I don’t know how you do this.

But, I have other ways to ”recreate” this containership info (and to support ordering-within-folders); and I don’t need to recreate the Firefox Bookmarks hierarchy.

I’m just trying to get clear on how your utility can help me.

Re: Exporting Bookmarks STRUCTURE, AS WELL AS content

DickB36 — Tue, 18 Apr 2023 13:39:31 GMT

I meant to mention that I need to identify the FOLDER PATH to Bookmark_A (and any other Bookmarks).

Re: Exporting Bookmarks STRUCTURE, AS WELL AS content

DickB36 — Tue, 18 Apr 2023 14:02:02 GMT

"Drop your JSON or JSONLZ4 file onto the box"???

When I click on the link you sent me, I get to a page with your source code (on GitHub).

When I select "Got o File," II get to a GitHub) Firefox Utilities page.

WHICH utility do I select? Where can I enter the fullpath to the .JSON file I want to test?

Re: Exporting Bookmarks STRUCTURE, AS WELL AS content

jscher2000 — Tue, 18 Apr 2023 16:00:42 GMT

@DickB36 wrote:
"Drop your JSON or JSONLZ4 file onto the box"???
When I click on the link you sent me, I get to a page with your source code (on GitHub).

You need to use the link I provided originally:

https://www.jeffersonscher.com/ffu/bookbackreader.html

or the Github-hosted version of the page:

https://jscher2000.github.io/Firefox-File-Utilities/bookbackreader.html

Re: Exporting Bookmarks STRUCTURE, AS WELL AS content

DickB36 — Tue, 18 Apr 2023 23:33:34 GMT

Lovely!

I can get to where I need to much more easily from your .TSV

Thanks!

Re: Exporting Bookmarks STRUCTURE, AS WELL AS content

DickB36 — Thu, 20 Apr 2023 07:58:03 GMT

The Firefox “Manage Bookmarks” utility (Library) has two “sections.”

1) The “Tree,” which shows our Folders (branches), and the bookmarks stored in each Folder

2) A composite listing of all our bookmarks (at the bottom of “Bookmarks Menu”

I only wanted bookmarks Title, URL and Folder info, so I only “peeled” the records from the “Tree” section.

The utility that Ischer2000 suggested produced TAB-limited (.TSV) file that extracted this information from a “Backup” copy of the Library (.json), which made it reasonably-easy to capture what I want.

https://www.jeffersonscher.com/ffu/bookbackreader.html

I ran into one problem in the process of converting the Firefox records to a data file:

In the TITLE field, non-priming (and “invisible” characters cropped up in some of these records (and the FOLDER PATH field for all records). These were easily trapped, and stripped-out. as part of the operation of transferring the .TSV info to a data-file; from where I can perform all of the additional conversion operatios I'll need.

Thank you, Ischer2000!

Re: Exporting Bookmarks STRUCTURE, AS WELL AS content

jscher2000 — Thu, 20 Apr 2023 22:00:03 GMT

In the Folder path, was the problem the use of right-angle quotation marks to indicate the nesting? You could revise the script to insert a different character.

https://github.com/jscher2000/Firefox-File-Utilities/blob/master/bookbackreader.html#L490

To download the files to use locally, click the green "<> Code" button here and grab the zip file:

https://github.com/jscher2000/Firefox-File-Utilities

Re: Exporting Bookmarks STRUCTURE, AS WELL AS content

DickB36 — Fri, 21 Apr 2023 20:20:44 GMT

Ischer,

No, the high-ASCII chars. did not include the right-angle marks (I knew that they were parto of what in JSON formatting is referred to as "children":[ (and as TypeCode:2) -- i.e., "folders" (or "nodes").

In the FOLDER PATH segment of the .TSVA string, "Bookmark Menu became "BookMarkA" then a space; then the angle-brackets.

In the TITLE segment, 2 or 3 non-printable chars. SOMETIMES appear at the beginning of the (TAB-delimited) string) -- and different ASCII chars further-on in the string.

This behavior was NOT consistent. But, instead of looking for a pattern in its occurrences, I just wrote a function that parsed the entire TITLE string and stripped-out the problem chars.

Otherwise, your utility saved me DAYS of work (thank you for that). Now I'm working on cleaning-up the occasional damaged record -- which SEEMS to be limited to te END of the FOLDER PATH string:

"Speeches;S" "Speeches;Sta" "Speeches;Stat" " Speeches;Statements"

These turned-up in 4 CONSECUTIVE bookmarks. The only possible explanation is that the over-all LENGTH of the block of chars. I retrieved from your file was LESS than the length of the over-all "reecord (upto the CR). I'll check; but it doesn't seem to be. that.

This corruption of the data seems to be more-extensive (I know that all of these records -- bookmrks -- are in the SAME folder).

Anythoughts?

Re: Exporting Bookmarks STRUCTURE, AS WELL AS content

DickB36 — Sat, 22 Apr 2023 16:45:39 GMT

Ischer,

All "corrupted" have been fixed. by making 2 corrections:

1) Increasedthe number of bytes read from your .TSV file (opened as a low-level file: i.e., not by aby app with a User-interface)

2) Increased the sizes of the data-file fields used to capture the TITLE, URL, and FOLDER PATH values "peeled" from the .TSV file.

Now, all of the info extracted from Firefox appears to be "good"!