Month: August 2016

New steps with bulk photo uploads

  1. We’ve proved the bulk upload for Artefacts. This is the most complicated section because of the many artefacts that have multiple images – this doesn’t usually apply to the photo archives, although there ARE a few instances like that.
  2. So uploading the rest of the Artefacts is just repetitve work – quite a lot of it.
  3. Need to discuss whether we have separate albums for each of the 11 lots of photos (If so, what should the albums be called?)
  4. The experiments below show that there are various photos in the archive files that do not have entries in the database.

Got as far as creating an XLS file from Table 1 with appropriate columns: “Archive Photographic Register Table 1.xls”

Noted that a few photos have double suffixes “.jpg.jpg” – don’t know whether this will matter.

Created item column as (Album no)_(Photograph no) – but needs to copy Value, not Formula!

Expanded multi-image lines

Transferred filenames to Column 1

Uploaded SWEHS 1.0 old exhibitions and appliances to Depot (mainly jpg files, one pdf) using Filezilla

Saved Archive Photographic Register 1 as CSV, uploaded to Depot using Filezilla

Opened Photo Albums – Import,waited for images to load. Unchecked the csv file, clicked Import button.
“88 single photos imported”

Imported the images but without the data.

Ticked csv file, clicked Import button.
(Maybe the tick isn’t just “remove when imported”, could be what tell it to import the csv file)

“All available custom data fields are in use. There is no space for Album No”

But I realised that I had put them into the Artefacts album – so I deleted them.

Created a new Album “SWEHS 1.0 Old Exhibitions and Appliances”

Deleted column Album No in xls file, also saved as csv.

Opened Photo Albums – Import,waited for images to load.
Set the Import Album to “SWEHS 1.0 Old Exhibitions and Appliances”
Left the csv file ticked, clicked Import button.
“88 single photos imported”

“All available custom data fields are in use. There is no space for Photograph No”

Deleted Album Title and Category in xls, csv.
Remaining columns are:
filename
id
item
location
date_made
desc
comments
copyright

Uploaded manded csv with Filezilla.

Clicked Import
“1 CSVs imported, 0 photos processed. 1371 photos skipped.”

In new Album, but still no data. What isn’t matching? Simple – the data file doesn’t contain the photos I uploaded!

Uploaded all the 3.1 photos instead (around 100) – note that most but not all of these are in the data file!
Also some filenames for photos have spaces in them…
Created a new album “Bristol Corporation Electricity Department”

Clicked Import
“Time out. 70 photos imported. Please restart this operation.”
Clicked Import again
“28 single photos imported. 1 CSVs imported, 75 photos processed. 1296 photos skipped.”
Data uploaded OK for the photos whose data was in the file.

In the data file, the 3.1’s go up to 077, and then the 3.2’s start. No data evident for 3.1 photos 079 to 100.

Copied the 3.2 files across (154 files). Re-imported the CSV file.

Three PDFs removed automatically.
CSV file not visible in Import, but clicked Import nevertheless.

“115 single photos imported.” – don’t understand where the other 50 or so went!

Copied CSV file across again several times – realised that it won’t show in Import screen if the original is open on the PC (because it’s then not importable)

Clicked Import
“1 CSVs imported, 170 photos processed. 1201 photos skipped.”

Most photos have data imported, but 3.2 photos 212 to 228 have no data – because the data file has no data for them!!!!!!

Experiments with bulk photo uploads

Summary of this very technical post:

These are the details of various experiments in bulk loading the massive amount of photos and data for the Artefacts. Some of the issues are to do with cleaning up the data from the Artefacts database. Others are about getting reasonable numbers of photos uploaded at a time, avoiding timeouts. At the time of writing the first 150 artefacts out of about 700 have been uploaded successfully with their data.

=====================

Importing artefacts photos:
Imported Access file into Excel
Removed all HTML tags (look for < and &)
Filled in blanks on Description column
Not sure how it will handle .doc files in the image column!
Expand multiple filenames manually

Plugin used is WP Photo Album, website http://wppa.nl/
Custom Datafields needed (set in the album in Table II J-10) are:
filename
id
item
date_made
model
desc
comments
manufacturer
location
copyright (not populated for artefacts)

Noticed JPG AND jpg – different cases in the suffix – don’t think this matters if the data matches the actual filename.

Took everything up to image 70, with some “multi-image rows” left in for test load
Zipped the photos and the CSV file into a single zip file.

Attempted to upload zip file testload04082016.zip (via Photo ALbums / Upload Photos / Box C)
Error 405 – not allowed. Assumed that this was because I had more than 20 photos.
Split into 4 parts, 20 – 20 – 20 – ***30*** photos

It imported all four files (even the 30 photo one), didn’t overwrite duplicate photos already in the Album (good), but didn’t import the descriptions for some photos on certain of the “multi-image rows” (not what I hoped, but it was worth trying!). Will need to reload these as single lines.

Tried separating lines out

Got Invalid header. First item must be ‘name’, ‘photoname’ or ‘filename’ (but I didn’t zip the file!)
Moved filename to column 1, put into zip file. Tried again, also with a photo included in zip file, but that didn’t work.

Couldn’t delete superfluous files from “depot” either.

Uploaded the “separate lines” CSV to depot using Filezilla.

Deleted all Artefact photos.

Put all 88 photos up to no. 70 into a new zip file and uploaded, but it only imported one photo. Probably uploaded the wrong file.

Tried agaiin, got 405 not allowed – timed out?

Used Filezilla to upload zip file to depot

“File test artefact load 04082016 no 2/SWEHS000057.zip is of an unsupported filetype and has been ignored during extraction” – not surprising, it was a zip file, but otherwise extracted 88 files. “1 Zipfiles extracted. 1 CSVs imported, 0 photos processed. 34 photos skipped.”

Used Filezilla to upload a new version of the zip file of photos to depot.

87 files imported, 1 already there.

Photos imported, but not the data.

Deleted all the photos again.

Put photos and csv file in a single zip.
Used Filezilla to move new zip file to depot.
Ran Import again.
Again, photos loaded but no data.

CSV file copied to depot, Import run.
Invalid header. First item must be ‘name’, ‘photoname’ or ‘filename’ – I’d imported the wrong one.

Right CSV file copied to depot, Import run.
Photos, and most – but not all – have data!

Need to work out which ones out of the following don’t have data, and why…

SWEHS000005a.jpg
SWEHS000005b.jpg
SWEHS000048b.jpg
SWEHS000068a.JPG
SWEHS000068b.JPG

The above are not in the data extract from the database, so not surprising that no data is uploaded.

SWEHS00020.jpg
SWEHS00029.jpg

These don’t have enough leading zeroes in the filename. Filenames are correct, checked against database. There are duplicate image files with one less zero! Solution – entries deleted from WordPress album.

Converted all artefact database to CSV and removed all except required columns – file: “artefacts 20160807.csv”

Expanded multi-image lines from 71 to 150.

Removed a couple of duplicate files with DC in filename.

Put images 71 to 150 and CSV into zip file: “artefact upload 71 to 150.zip”

Transferred zip file to Depot “/swehs/wp-content/wppa-depot/paul.hulbert@gmail.com/” using Filezilla.

Removed old CSV file from Depot.

Photos / Import

It extracted the photos, showing images.

Clicked Import. “Time out. 76 photos imported. Please restart this operation.”

50 photos left in Depot. Clicked Import again. “50 single photos imported”

Album Admin shows no data imported for the new photos.

Copied master CSV file “artefacts 20160807.csv” to Depot using Filezilla.

Imported – “1 CSVs imported, 193 photos processed. 1863 photos skipped”

All photos appear to have data attached, no duplicates – success!

© 2017

Theme by Anders NorenUp ↑