[VUFIND582] ALLOW INDEXING RESERVES FROM CSV CREATED 18MAY12 UPDATED

[VUFIND-582] allow indexing reserves from CSV Created: 18/May/12  Updated: 25/Oct/12  Resolved: 25/Oct/12

Status:

Resolved

Project:

VuFind

Component/s:

None

Affects Version/s:

None

Fix Version/s:

1.4, 2.0RC1

Type:

Improvement

Priority:

Minor

Reporter:

Laura Hild

Assignee:

Demian Katz

Resolution:

Fixed

Votes:

1

Labels:

None

Attachments:

index-reserves-file-vf1-v2.patch     index-reserves-file-vf1.patch    

Comment by Laura Hild [ 18/May/12 ]

This patch allows you to specify a CSV file with index_reserves.php -f. -d lets you change the delimiter.

Comment by Filipe M S Bento [ 18/May/12 ]

Hi, Steven!

Many thanks for the patch; actually since in my former services (Library) they stopped using ALEPHs Course Reading Module (xxx30) and instead publish .xls files in the Library Portal, this comes in handy.

Just to clarify: 'INSTRUCTOR_ID', 'COURSE_ID', and 'DEPARTMENT_ID' have to be created/present in the ILS, right, I mean their data? This just overrides the Bib IDs associated with each Course that otherwise would be read from ILS API/web services (or not, in our case, although most of the rest info is there -- previous mentioned fields).

All the best (and thanks, once again),

Filipe

Comment by Laura Hild [ 21/May/12 ]

The patch allows you to populate the reserves index from a file in lieu of calling the ILS driver. If you're running reserves in Solr/indexed mode, I believe VuFind will ignore your ILS driver's reserves functionality.

It avoids the issue of instructor, course, and department IDs by setting them equal to the values. An earlier implementation used a sectioned CSV file in which IDs were specified, but that seemed awkward and unnecessary. "[VuFind-Tech] course reserves IDs" may be enlightening regarding this.

Comment by Filipe M S Bento [ 21/May/12 ]

Dear Steven,

Yes, I've seen that message prior to post the above comment. So, if I understood correctly, IDs are the names themselves ('INSTRUCTOR_ID', 'COURSE_ID', and 'DEPARTMENT_ID'), not codes stored in the ILS Course Reserve database, that's it?. OK!

Will try ASAP.

Many thanks,

Filipe

Comment by Filipe M S Bento [ 21/May/12 ]

Ok, just to clarify and help others that may want use this method, the CSV file has to be UNIQUE (else it will overwrite all entries that are there -- very handy to change the Course Reserves each Semester), for instance

000220309;Algoritmos e Complexidade;Joaquim Madeira;DETI
000158140;Algoritmos e Complexidade;Joaquim Madeira;DETI
000223427;Algoritmos e Complexidade;Joaquim Madeira;DETI
000158141;Algoritmos e Complexidade;Joaquim Madeira;DETI
000158142;Algoritmos e Complexidade;Joaquim Madeira;DETI
000217359;Algoritmos e Complexidade;Joaquim Madeira;DETI
000170185;Algoritmos e Complexidade;Joaquim Madeira;DETI
000113594;Análise Complexa;Maria Isabel Jordão Cação;DMat
000001135;Análise Complexa;Maria Isabel Jordão Cação;DMat
000176199;Interação Humano-Computador;Paulo Dias;DETI
000214208;Interação Humano-Computador;Paulo Dias;DETI
000233027;Interação Humano-Computador;Paulo Dias;DETI
(....)

In which:
column 1: record_id (that must exist in solr/biblio index)
column 2: COURSE
column 3: INSTRUCTOR
column 4: DEPARTMENT (attention: does not allow long names, so in my example, used the initials of the Department)

If the name of the file is CourseReadind2012_2sem.csv, it can be loaded (after activating reserves solr index -- see config.ini how to do it => restart vufind)

from /util run (having the file with data to load there or elsewhere):

$ php index_reserves.php -f CourseReadind2012_2sem.csv -d ';'

Check /Search/Reserves and you should have as many lines as Courses (with links to the same list in the entries of Department, Course and Instructor -- strange enough not in the number of items... going to change that).

File must be UTF-8 if it has special chars in any of the names.

All the best,

Filipe

PS: please correct if anything is not ok!

--------------------------
Filipe Manuel S. Bento | http://about.me/filipeb
Computer Science Specialist * Electronics and Telecommunications Eng. (UA)
MSc Electronic Information Management (USheffield, UK) * ICPD Doctorate Candidate (UA | UPorto)
PhD Researcher (UA/CETAC.Media) grant by FCT - Portuguese Foundation for Science and Technology
President/Chair of USE.pt Steering Committee (Portuguese Ex Libris Users’ National Association)

Comment by Laura Hild [ 21/May/12 ]

> IDs are the names themselves ('INSTRUCTOR_ID', 'COURSE_ID', and 'DEPARTMENT_ID'),
> not codes stored in the ILS Course Reserve database, that's it?

VuFind's indexed reserves functionality asks that IDs be supplied for departments, courses, and instructors, and that reserve records refer to departments, courses, and instructors by their ID, which is how an ILS driver supplies them. In order to avoid complicating the file format, the patch does not require IDs: if the instructor's name is "Willy Wonka", the instructor's ID, as provided to VuFind's indexed reserves functionality, is also "Willy Wonka". Since VuFind does not appear to contact the ILS driver about reserves when indexed reserves are enabled, the IDs the ILS driver would have provided, or those used by the ILS itself, which is also not contacted regarding reserves, are ignored.

> the CSV file has to be UNIQUE (else it will overwrite all entries that are there

I'm not sure what you mean by "unique". You should be able to specify -f multiple times, and it should be as though the files were concatenated. It is true that a run of index_reserves.php that successfully acquires data will wipe the index before adding the data it acquired, with or without the patch.

> In which: [...]

Yes. The intention was that they be ordered in roughly decreasing order of specificity.

Comment by Filipe M S Bento [ 21/May/12 ]

Hi Steven.

True, didn't try -f multiple times in the same command line (didn't know was possible), but every time I indexed them the others that were there... puff... disappear!

So UNIQUE is not mandatory, but if have this in database or some sort of management unit (for instance, via web interface), you will end up with an unique file exported from SQL Server, MySQL, Access what whatever,,,

Again, thanks for the patch, although I am going to give directions / advise Library stuff to use dynamic public lists as Course Reading list to Instructors so they pass their URL to their Course students >> it hasn't to be static, and the instructor can add resources as he/she finds them in the system and are of an higher level....

All the best from Aveiro, Portugal (and an installation that is still in progress of implementation / more to come -- unfortunately a one man show, me, in what it comes to implement / program and most of want the system to be).

FB

PS: sorry for the specs about the entry file(s), but since you haven't given a sample of one, had to do several trial runs until get it right, so that is why I shared those clues about it.

Comment by Laura Hild [ 22/May/12 ]

> sorry for the specs about the entry file(s)

Don't be. This patch needs documentation, despite my lack of feeling documentary when I posted it. :-) When I do feel documentary, I'll update the patch.

Comment by Filipe M S Bento [ 24/May/12 ]

"This ticket adds support for building the Solr reserves index from a CSV to allow it to work with ILSes that do not allow reserves data to be loaded directly through the driver. Demian plans to test and commit in the near future."

Save yourself that work (I know you have to, it's better to ensure full compatibility)... Just to say that is working ok: see it running in my staging install.

Btw, I would make it another util file, different from the present one; I mean no need to have that switch in any .ini file: if someone needs to load Course Reading reserves would call this different version instead (one with a name "from_ILS.php", other "from_CSV.php" + a sample .csv file (in trunk's).

Filipe

Comment by Demian Katz [ 25/May/12 ]

I think it is advantageous to implement this as a command-line switch rather than a separate file since it helps avoid duplicate logic in the code (not to mention proliferation of files in the util directory). I like grouping related things into compact units. It would be pretty easy to write a shell script wrapper around the single file if you really needed to have separate commands for the two actions.

Comment by Filipe M S Bento [ 25/May/12 ]

Brilliant!

Filipe

Comment by Demian Katz [ 25/Oct/12 ]

I've expanded this a bit (adding help messages and a -t parameter which can be used to specify a template for field order) and committed it as r5978. See attached patch for the updated diffs.