UnitedCardists.com - The forum for Cardistry, Magic, Custom Decks and All Things Playing Cards.

Would love to be involved but still not able to join.

shaneriley wrote: ↑Tue Mar 19, 2024 9:49 am I wasn't sure about uploading the images, since each user granted P52 with a license to use them and I'm not affiliated with P52.

Would you be able to zip and send those images to me? I already have a public Google drive with some images scraped from two user collections but they're nowhere near complete. It should be easy to remove the duplicates and upload the rest for others, particularly Manu, hsbc, and the like, to use to rebuild from.

With your .json file and these images, there's some significant potential for rebuilding and improving upon the entire P52 database. I can see people with technical knowhow automating it all, although manual entries might be less error prone(?)

Here's the drive in question: https://drive.google.com/drive/mobile/f ... sp=sharing

kevork wrote: ↑Wed Mar 20, 2024 2:18 am Would you be able to zip and send those images to me? I already have a public Google drive with some images scraped from two user collections but they're nowhere near complete. It should be easy to remove the duplicates and upload the rest for others, particularly Manu, hsbc, and the like, to use to rebuild from.

With your .json file and these images, there's some significant potential for rebuilding and improving upon the entire P52 database. I can see people with technical knowhow automating it all, although manual entries might be less error prone(?)

Here's the drive in question: https://drive.google.com/drive/mobile/f ... sp=sharing

I'll compress it all and upload it tonight. I don't see a way to view the file count in your archive, but my local one has 12,099 images.

My deck images are here: https://www.dropbox.com/scl/fo/itaq414z ... kme6f&dl=0

should be 1500 or so decks worth

shaneriley wrote: ↑Wed Mar 20, 2024 11:30 am

kevork wrote: ↑Wed Mar 20, 2024 2:18 am Would you be able to zip and send those images to me? I already have a public Google drive with some images scraped from two user collections but they're nowhere near complete. It should be easy to remove the duplicates and upload the rest for others, particularly Manu, hsbc, and the like, to use to rebuild from.

With your .json file and these images, there's some significant potential for rebuilding and improving upon the entire P52 database. I can see people with technical knowhow automating it all, although manual entries might be less error prone(?)

Here's the drive in question: https://drive.google.com/drive/mobile/f ... sp=sharing

I'll compress it all and upload it tonight. I don't see a way to view the file count in your archive, but my local one has 12,099 images.

My folder had 11,809 images. With Gandalf's image scrape copied into the folder, that number is now 13,720 images. I've uploaded them into the drive.

The Wayback Machine's latest snapshot of P52 on February 4th, 2024 shows P52 having 24,956 unique decks in the archives. [Source]
If we're assuming 2 images per deck, that's 49,912 images. So with 13,720 images, we have about 27% of the image data. With 17 lines of code per deck metadata and the json file being 63,227 lines, it seems to be 3,719 decks, so about 15% of P52's total metadata. It's certainly a foundation for a rebuild.

Since P52 beautifully named the image files with the corresponding deck number, they can be cross-referenced with the json file.

Here you can see an example of the process and 3 of the 4 possible scenarios, with the fourth scenario not illustrated here: both image/metadata missing.

• "Deck 5944" has the corresponding image and metadata.
• "Deck 5947" has the metadata but no images.
• "Deck 5952" has the image but no metadata.

Here's my archive of images. I had made a quick web app that combined this image set with the data I scraped from Wayback, and a large number of decks don't have images. Still, they might be useful for requesting new photos of the decks.

https://www.dropbox.com/scl/fi/90ixxqsu ... k3d5m&dl=0

We're up to 600 decks added

Anyone who has access, I'd appreciate any help adding decks! If you don't have a scanner, Henry at SoCal said we can use scans on his site (see the past auctions for a good place to start?)

Shoutout to Timmargh for adding my white whale

hsbc wrote: ↑Thu Mar 21, 2024 3:39 pm [...]

Shoutout to Timmargh for adding my white whale

*double-checks locks; cocks gun*

Can't be too careful.

More requests:

Tongheng > Tongheng (KWPCC)
Kings Wild Project > Kings Wild Project (KWP) -- same thing with t11, TWI, S17, AoP please
Deckidea > Jason Cheng (Deckidea) -- this is in Artists

On the add deck upload page, you have instructions under some of the field names -- could you add one under release date? Something like:

"A year printed on the tuck or cards overrides any date of funding or delivery. Note any differences in the description."

And a quick mockup idea:



Maybe swap out the stats on the right with a logo and links?

I also think fields for primary and secondary colors would be awesome

Timmargh wrote: ↑Thu Mar 21, 2024 5:12 pm *double-checks locks; cocks gun*
Can't be too careful.

This sparked a suggestion. An option to anonymously upload a deck, but you'd still need an account to upload; it just wouldn't show up publicly. This can be a protective measure. Ideally, if a user does not check "Upload Anonymously" they should still be able to remove their username from the upload retroactively.

EDIT: 214 additional images were added to the Drive folder thanks to ones Shane had scraped that weren't accounted for previously.

An idea for the front page -- users vote each week for a producer or artist or category, and then rank the winning creator's decks. Then the top five or so decks are displayed on the front page as "community picks" or something Artists can't be picked more than once in a... while? We want to highlight different cards each week is my thinking here

Aaaaand P52 is back up Maybe? Minutes later, it's not loading once again

I wrote a quick scraper to grab all deck info, and it's working through it now. I'm up to 118. Hoping to then download the front/back deck photos, and finally the gallery photos. Will then update the Internet Archive entry to include the data.

I might also create a direct clone of the site, since I've already set up a basic API using the data and set up user auth. Shouldn't be too much more work to set up the deck POST action. From there, I could add the features I've always wanted in P52.

Will update everyone here when the archive is complete, or when P52 goes down again and I've posted as much as I've obtained.

shaneriley wrote: ↑Fri Mar 22, 2024 6:35 pm I wrote a quick scraper to grab all deck info, and it's working through it now. I'm up to 118. Hoping to then download the front/back deck photos, and finally the gallery photos. Will then update the Internet Archive entry to include the data.

I might also create a direct clone of the site, since I've already set up a basic API using the data and set up user auth. Shouldn't be too much more work to set up the deck POST action. From there, I could add the features I've always wanted in P52.

Will update everyone here when the archive is complete, or when P52 goes down again and I've posted as much as I've obtained.

Any tips or guide on how someone can go about this, Shane? I'm a novice but so curious. Not necessarily the website cloning part although that sounds awesome too, but moreso the data scraping portion.

P52 just loaded for me - didn’t log in yet - but I’m already dancing

It's back up

kevork wrote: ↑Fri Mar 22, 2024 7:03 pm Any tips or guide on how someone can go about this, Shane? I'm a novice but so curious. Not necessarily the website cloning part although that sounds awesome too, but moreso the data scraping portion.

I write mine in Node. The main code uses the core http Node package and cheerio for constructing a DOM and plucking data from the page using jQuery-like selectors. It requires a good knowledge of JavaScript programming to build. The one I built runs in 100-deck stretches, with 500 decks per JSON file. It's not super-polished, but it works. After that runs, I'll write a similar script to download the images.

I can't really give you pointers other than to learn JavaScript. It comes in handy more often than any other programming language these days.

shaneriley wrote: ↑Fri Mar 22, 2024 7:36 pm

kevork wrote: ↑Fri Mar 22, 2024 7:03 pm Any tips or guide on how someone can go about this, Shane? I'm a novice but so curious. Not necessarily the website cloning part although that sounds awesome too, but moreso the data scraping portion.

I write mine in Node. The main code uses the core http Node package and cheerio for constructing a DOM and plucking data from the page using jQuery-like selectors. It requires a good knowledge of JavaScript programming to build. The one I built runs in 100-deck stretches, with 500 decks per JSON file. It's not super-polished, but it works. After that runs, I'll write a similar script to download the images.

I can't really give you pointers other than to learn JavaScript. It comes in handy more often than any other programming language these days.

I know some of these words

bdawg923 wrote: ↑Fri Mar 22, 2024 7:48 pm I know some of these words

Haha, exactly what I was going to say!

Get out of my head, Charles!

I’ve been neck deep in node for the last 10 years - I can help you hack through it

And if anyone wants amateurish PHP help then give me a shout.

To download all deck images, assuming there are 26,742 decks in the database:
To download gallery images:
Linux only, I don't know how to do it using Windows tools.

This folder is 11.5Gb

Apparently it was not Alex who rebooted P52...

montenzi wrote: ↑Fri Mar 22, 2024 11:01 pm To download all deck images, assuming there are 26,742 decks in the database:

Code: Select all

wget -m https://portfolio52.com/files/700x500/deck_{1..26742}_back_img.jpg
wget -m https://portfolio52.com/files/700x500/deck_{1..26742}_front_img.jpg

To download gallery images:

Code: Select all

wget -m https://portfolio52.com/files/700x500/gallery_item_{1..20000}.jpg
wget -m https://portfolio52.com/files/700x500/gallery_item_{20001..40000}.jpg
wget -m https://portfolio52.com/files/700x500/gallery_item_{40001..58662}.jpg

Linux only, I don't know how to do it using Windows tools.

This folder is 11.5Gb

Nice to know it's that easy. I tried it on Windows using the wget binaries from: https://eternallybored.org/misc/wget/ but the link you provided wasn't able to automate it in the ways you were probably able to on Linux. Can anyone upload the 11.5Gb file to be downloaded?

Thank you!

Timmargh wrote: ↑Thu Mar 21, 2024 5:12 pm

hsbc wrote: ↑Thu Mar 21, 2024 3:39 pm [...]

Shoutout to Timmargh for adding my white whale

*double-checks locks; cocks gun*

Can't be too careful.

LOL there are only 15 of it - good luck with that hsbc

kevork wrote: ↑Fri Mar 22, 2024 11:37 pm Nice to know it's that easy. I tried it on Windows using the wget binaries from: https://eternallybored.org/misc/wget/ but the link you provided wasn't able to automate it in the ways you were probably able to on Linux. Can anyone upload the 11.5Gb file to be downloaded?
Thank you!

I'll disable links in a couple of days.

https://old.montenzi.nz/p52-700-boxes.tar (5GB) - new file with boxes only - UPDATE
https://old.montenzi.nz/p52-gallery.tar (19GB) - gallery images - UPDATE

P.S. I'm not sure what my server speed limit is, but I'm downloading it now at 100 Mbps, and my local connection is 300 Mbps.

It seems that the original gallery photos are in another folder, which is 20 GB in size. Yes, I downloaded this folder to my server.

The "gallery_item" files in the 700x500 folder are cropped to fit the size of the box, and I'm not sure of their purpose. Backs and fronts are still there.

I updated the file to backs and fronts only, removed the old file and added a new "gallery" file.

Portfolio 52 is back up. I hope this doesn't derail this site. It's nice to have options

montenzi wrote: ↑Sat Mar 23, 2024 12:59 am [...] my local connection is 300 Mbps.

I had 100Mbps in my old place about 10 years ago, but now I'm lucky if I get 25.

shaneriley wrote: ↑Fri Mar 22, 2024 6:35 pm Will update everyone here when the archive is complete, or when P52 goes down again and I've posted as much as I've obtained.

How'd it go?