Re: [Gnumed-devel] how to do intended_reviewer / requestor? (was: HL7 test results import.)

[lkcl]

ok, this one deals with "semantics", "information" and "meaning".  it's
assumed that syntactical and grammar errors have already been taken care of,
by this point (import_log, whatever).


Jim Busser wrote:
> 
> An editing widget would need only a few fields (not 50-150) and these
> could simply added to the unmatched table:
> 
> fk_import_id (foreign key to the source file, as referenced in import_log)
> 
> message_content (the raw, unparsed message NOMODIFY)
> 
> automatch_result (suggest the script populate this with one among:)
>       matched <-- to patient and provider; invisible to user; proceed to
> auto-create
>       patient? <-- await manual matching *or* approval of creation
>       provider? <-- infrequent, unusual routing *or* a praxis SNAFU
>       ?? <-- rare (but possible) major source SNAFU
> 
> patient_identifiers (parser populates using HL7 PID 001 … 013)
>       --> e.g. ||||SMIFF^ARFUR||19450928|M|||^^SURREY^^BC||585-1510
> 

... ok, so great - you parse the patient identifiers, great, but what about
the rest of the context?  is the staff-admin expected to view the _rest_ of
the raw HL7 data and actually understand it??  surely it would be better to
use the exact same clin.lab_results + clin.test_data dialog boxes to review
the data, yes?



> provider_identifiers (parser populates using HL7 OBR 016 | OBR 028)
>       --> e.g. 22333^MEDIC^IAN^TEST|00000^NEWDR^JAMES~H09999^DISNEYLAND^CLINIC
>       (note: ORC 012 could be used in place of OBR 016 as these are the same)
> 

again, same thing: _partial_ decode of the HL7 data is extremely unfair to
the ordinary average person, whose mind now has to cope with the rest of the
HL7 data in raw format (bar the provider identifiers and patient
identifiers, even if those _can_ be matched to records in gnumed patient and
staff records)



> if so - then basically, this would be a mini-project on its own, and
> i'm not convinced of its value... and if you're going to present "lab
> data", why not spend the time creating a gnumed lab_request and gnumed
> test_results viewer, instead of an HL7-specific viewer?

I think the above (with or without refactoring) is sufficiently general to
achieve what you suggest (?)


honestly speaking? it's general to the point of being inadequate: it doesn't
solve the problem of making the staff-admin's life easier, quicker and
simpler, by getting them away from the "raw" HL7 data as fast as possible,
and into easy-to-view tabular format as soon as possible.



>> also, by simply "auto-creating" a tag-identifiable patient and
>> "auto-creating" a tag-identifiable doctor (as staff) in cases where
>> automated disambiguation is not possible, the import _will_ succeed.
>> 
>> everything else is on auto-create, and goes down as "lab data", so
>> why not patients and doctors too?
> 
> Good question. Short answers:
> 
> 1) when patients did not *seem* to exist,
>       - they may already exist
>       --> in this case, the auto-created persons will have to be merged
>       - or, even when they did not yet exist, they should sometimes not be
> created
>       --> therefore better that creation be user-approved
>               --> it can then be done on the next iteration
> 
 * in "already exist" case, my view is that the most appropriate tool to
assist in that merging process is the exact same dialog box that is showing
the clin.lab_request and clin.test_data tables _not_ a "separate" tool which
is a) inadequate b) duplication of effort (wrt the
clin.lab_request+clin.test_data dialog box) if it _is_ made "adequate" c) is
specific to HL7 whereas if you make it the same tool then that same tool can
be used for *other importers as well*.



>       - or, even when they did not yet exist, they should sometimes not be
> created
> 
 there is no logical reason (at least not one clearly and explicitly stated)
why this should be the case...


>       --> therefore better that creation be user-approved
> 
 ... thus this justification comes out of nowhere.


>               --> it can then be done on the next iteration
> 

 ah.

 a) prior to which the admin/staff have been forced to read raw HL7 in order
to help correctly ascertain who the patient really is, and their brains are
now in knots

 b) and now the situation is a mess, already, because the prior iteration
terminated half-way through a message (which let's say it happens to be an
"update" HL7 message); some of the data in the clin.lab_request has been
overwritten already by that point.

 c) coding to mitigate the b) situation above is _itself_ messy, requiring
database table locking and/or resulting in queries that may *or may not* be
adequate to determine, in advance, whether the insert/update is going to
succeed, and even if you do get it right, you've still added a layer of
complexity to gmHL7.py which is, in my opinion given the alternative
solutions (one of which is auto-create-of-everything), entirely unnecessary.



> 
> 2) when providers did not seem to exist
>       - something is usually wrong
> 

... and it is my view that the best way to determine what is wrong is to be
able to see the clin.lab_request data and the clin.test_results using the
exact same dialog boxes as are used to show "right" data, except with "big
red marks" easily, simply and clearly identifying that "wrongness".




> Doctors are cautious (hopefully you would prefer that they be this way
> <grin>) so let me offer some prototypical, conservative doctor-responses
> to some scenarios, and see to what extent programmer's rebuttals can "win
> over" the arguments:
> 
ok :)  it's the same one, actually not so much a "programmer" viewpoint as
one of a "designer / architect" viewpoint, thinking about the work-flow. 
i'm _not_ "just a programmer".  if i was, i would have implemented precisely
and exactly what you asked for.

and the viewpoint is repeated a number of times for clarity, and i'm stating
it repeatedly for the sake of completeness (and also there are some other
points).  i have some additional ideas though, as a result of reviewing
this: i'll do a separate message after this.


> ******************************
> 1) Praxis downloads a file containing messages on multiple patients NOT
> EXISTING in the praxis, ordered by or copied-to NO PROVIDERS in the
> praxis. On surface, this sounds like it should have to be some colossal
> mistake, but it could be *possible* where some file got written by the
> path lab into the wrong inbox, say maybe as a result of a corrupted index.
> In such a scenario, it would be *far* more sensible to reject the import
> file (i.e. batch-tag all its imported but unmatched messages for deletion)
> instead of having to find inside GNUmed multiple auto-created patients and
> provided who really never ought to have been created.
> 
> --> Therefore, for messages in which no provider can be matched, do not
> auto-create patients.
> --> Praxis "IT support" may need to delete all messages that were imported
> into the staging table from this source file
> 
a) " In such a scenario, it would be *far* more sensible to reject the
import file" - this requires that separate and additional logic be added
which does a complete review _before_ the message is added to the
clin.lab_request + clin.test_data tables.  the database will require locking
during that time, so as to prevent race conditions and the risk that extra
patients get added etc. messing up the actual add.  or, worse, pre-parse the
message once, get the info (patients etc.), then pass that on to the 2nd
walk-through, whoops, what happens if the patient record was deleted or
merged during that time?

b) " In such a scenario, it would be *far* more sensible to reject the
import file" - you haven't stated _why_ it would be "more sensible".  i've
not seen a detailed solution / scenario described which supports this
statement, and the one that's been given (create a generic "import log"
table) i've described the flaws and limitations of that approach, already
(namely that the "import log" idea is ideally and perfectly suited to
"syntax errors" but completely inadequate for these much more complex
"semantic" errors).

c) "--> Praxis "IT support" may need to delete all messages that were
imported into the staging table from this source file" - what is "wrong"
with deleting clin.lab_request, clin.test_data and "auto-created patient"
and "auto-created staff" records associated with same?  especially if it's
just a "delete" button on the dialog box in which
clin.lab_request+clin.test_data records are being presented, in which in big
bold red letters the patient (and perhaps provider as well) are marked as
"fake"?  oh, and the same "delete" button also "deleting all messages that
were imported into the staging table from the source file" at the same time?

what's wrong with that?



> 
> ******************************
> 
> 2) Praxis downloads a file containing messages on multiple patients who
> are either auto-matched or manually-matchable to patients in the praxis,
> BUT no matching provider exists. This can happen in two situations:
> 
> a) the praxis imported the results for this provider *without* the praxis
> administrator having created the staff (via GNUmed GUI). This should be a
> RARE-IF-EVER occurrence but, even if it did happen,
> 
agaaain, you're missing the point that the praxis administrator will be
forced to read the raw HL7 data in order to determine who that provider
really is.  my point is that it is much easier to have, on the exact same
clin.lab_request+clin.test_data dialog box a "warning! missing provider!"
button, behind which the missing provider, *having already been created* is
turned into a "real" staff member.


> 
> --> it's better that the administrator simply create the missing provider.
> 

why?  describe the work-flow to me.  describe the steps required, bearing in
mind that will require coding to get from here to there.  i think you'll
find that comparing the steps required and the amount of coding required to
implement each of them, the auto-create-fake-provider path is one which is
much a) easier to code, taking less time b) less code, taking less time c)
easier on the praxis administrator anyway!



>  Simply export any unmatched records, re-run through the importer, and
> these records will then auto-resolve.
> 

yes, with the disadvantage that the praxis administrator was forced to
review raw HL7, in a separate work-flow dialog no less (which is one of the
other reason why i don't like the idea), and is forced to do a "re-run".

whereas if the data's been auto-imported, the dialog box in which
clin.lab_request+clin.test_data is presented, they just click a button
(marked in red), select "make this fake provider a real one please", job's
done.  no re-run of importer, no reviewing raw HL7 data, not even an "import
log" table is needed to be involved, consulted, reviewed (or even added to
gnumed!)



> b) the ordering provider directed (or the patient requested that) a copy
> go to the praxis, but not to a specific provider within the praxis. This
> assumes that the praxis may have its own identifier, which may or may not
> be true for any path lab system. Supposing that this is possible,
> 
> --> the staff must decide which provider should inherit responsibility
> 
ok, so, again, the praxis administrator is forced to read raw HL7 data to
ascertain that; they're forced to do so in a _different_ dialog box from the
one presenting clin.lab_requests+clin.test_data in an easy-to-read format;
forced to re-run the import... wait... actually, there's no place in the
database to do this _at all_.  there is in the case of patients, because
there is room in the incoming_data_unmatched table for a
"fk_identity_disambiguated" record... but now there's really a need for a
many-to-one relationship because there will be multiple OBRs?  on top of
that you need to associate the "responsible provider" with the HL7 message
_as well_?  so now you need partially-decoded HL7 messages to be
cross-associated and that cross-association remembered across a complex
multi-stage workflow... achh it's a mess.   come on, jim :)

... alternatively, in the clin.lab_requests+clin.test_data dialog box, next
to the "big red warning" is a button "select alternative reviewer", list of
staff comes up, bam, record changed, big red warning goes away, job's done.

you see how much simpler that is?  i'm making these recommendations not so
much based on "how much code there is" but on "workflow" - to make the job
of the praxis administrator easier.  that's what technology is supposed to
be for - to make the average person's life easier not harder, and it's
something i think about a great deal.



> ******************************
> 
> 3) Patient is not matchable, but a provider is matchable.
> 
> Two main possibilities here:
> 
> a) a praxis provider was incorrectly inputted at the lab
>       å(the message should have gone to someone else; I have gotten these!)
>       … we do not want to create this patient
>       … we want to reject them, with a comment in the unmatched record
> 

 :) whoops.

 ok, then... big red button, push it - the fake-patient entry gets deleted,
the clin.lab_request entry(ies) get deleted, likewise the clin.test_data and
so on.  one button, one input-field, job done.

 remember, it's not _actually_ even a real patient: it's actually just an
entry into identity and names, with a dem.clin_ext_id_type of "lab data".

 but - thinking about this some more.... there iiis a little bit extra
work... let me think... i have mentioned this before, it would be a good
idea to cross-reference the original import data (originally from
incoming_data_unmatched) with the actual clin.lab_request and the actual
clin.test_data but that means having a "incoming_successfully_matched_data"
table, by which the original data can be moved _back_ to
"incoming_data_unmatched".

bit messy, but certainly a damn sight easier than the alternatives.



> b) praxis provider *was* correctly identified; in this case, the patient
> was:
>       … existing already in praxis, but was a failed match, *or*
>       … was cared for outside the praxis (but still reasonable to create)
> and/or
>       … is pending to join and be cared for in the praxis (still reasonable to
> create)
> 
> So 2a really only needs the affected messages to be reprocessed,
> Whereas 2b and 3b need the user-inputted data to "improve" the processing
> of the messages on the next iteration.
> 

this assumes that the work-flow which involves forcing the praxis
administrator to review raw HL7 data is acceptable, along with the
associated work-flow implications which i've covered above.



> So what we need to decide is:
> 
> i) which messages to "reprocess"?
>       anything that was modified after creation?
>       extend to include all messages from any one source file?
> 
"reprocessing" at the "semantic" level is exactly the situation i'm
recommended be avoided, because i can see clear as daylight that any other
workflow is complex, and the coding hairy.

i _would_ say "avoided at all costs", but actually the cost is not that
great.



> ii) how to incorporate user-inputted data into an "improved"
> (no-longer-original) message?
> 

 preeeeciselyyyy.  yesss.  this is _the_ key question.  "rewriting" the HL7
data is out-of-the-question.  not least because you then have a nightmare
audit trail on your hands.  so it would have to be done through "extra
tables" somehow.

and what better tables to use than dem.identity, dem.names, dem.staff,
dem.clin_ext_id_type, clin.lab_request, clin.test_data and
incoming_data_unmatched (and a new one incoming_data_matched)?

you see what i'm getting at?  if you'd like me to actually _do_ an
"improved" (no-longer-original) message system, _guess_ how i would
implement it?  not by modifying the original HL7 data - i'd implement it
by... yes, you guessed it: duplicating gmPerson.py, gmPathLab.py and so on
plus other associated code, duplicating dem.identity, dem.names, dem.staff
and clin.lab_request, clin.test_data tables and associated views, then
"after successful match", just copy the entries from the duplicately-named
tables into their corresponding "live" ones, and delete the entries from the
duplcately-named tables.

 so in this way, there would still _be_ auto-created-fake staff and
auto-created-fake patients, but they'd be in duplicately-named tables
separate and distinct from dem.names etc. etc.

 but that would be a silly thing to do.

 remember - all you have to do to "exclude" fake-patients and fake-staff
from being seen in the "standard" gnumed system is to modify the views
v_patient and v_staff to have a LEFT JOIN against dem.clin_ext_id_type of
"lab import person" and "lab import staff" respectively: you then add an
argument to get lists of patients which defaults, under normal
circumstances, to that field view being NULL.

 under these circumstances, to all intents and purposes, those "fake
patients "and "fake staff" no longer exist, and will _only_ appear under the
"big red warnings" in the (as-yet-to-be-written) clin.lab_request and
clin.test_data dialog boxes.

remember also: there _is_ no security risk with creating "fake staff"
because i have _not_ called the stored procedure "create_user".  there _is_
no username+password, the "fake staff" entry _has_ no login.  to turn that
"fake staff" into a "real practitioner", all that would be required would be
a minor alteration to gmStaffWidget to take into account the fact that the
staff table entry already exists, and to change (or delete) the
dem.clin_ext_id_type.

l.

p.s. i'll describe an alternative idea in a separate message: it involves
modifying the gnumed tables, though.  right now, the above proposal - which
i have actually successfully implemented - works by sitting *within* the
existing gnumed table structure.

-- 
View this message in context: 
http://old.nabble.com/Re%3A-how-to-do-intended_reviewer---requestor--%28was%3A-HL7-test-results-import.%29-tp29385122p29388357.html
Sent from the GnuMed - Dev mailing list archive at Nabble.com.