Sql interface lfs by eloise-nebula · Pull Request #21 · slaclab/slac-db

eloise-nebula · 2026-03-09T23:46:42Z

Added Oracle and AIDA DBs along with ways to generate them.

Database structure is defined as follows:

Lines 106 to 126 in de7a98f

    
           def _init_db(uri=None): 
        
               global _meta 
        
               if uri is None: 
        
                   uri = _oracle_uri() 
        
               uri = _db_type_prefix(uri) 
        
               schema = { 
        
                   "elements": { 
        
                       "Area": "str 64 nullable", 
        
                       "Element": "str 64 primary_key", 
        
                       "Control System Name": "str 64 nullable", 
        
                       "Keyword": "str 64 nullable", 
        
                       "Beampath": "str 64 nullable", 
        
                       "SumL (m)": "float 64 nullable", 
        
                       "Effective Length (m)": "float 64 nullable", 
        
                       "Rf Frequency (MHz)": "float 64 nullable" 
        
                   } 
        
               } 
        
               _meta = pykern.sql_db.Meta( 
        
                   uri=uri, 
        
                   schema=schema 
        
               )

uri defines the access method and location of the database.
schema defines the database columns and their data types. This is not a complete list for the Oracle db.

An interesting thing to note here is the global _meta variable defined at the top of the module. I'd never seen this pattern before looking through the slicops db implementation.

First, note that _meta is contained in the module's namespace. Code that imports slac_db.oracle can only access _meta with slac_db.oracle._meta. _meta tells SQLAlchemy how to access the database, and creates a session. The programmer using slac_db.oracle doesn't need to know about the _meta object, so we can hide it as an attribute of the module. Now we don't need a database_reader class to manage database location, format, access method, etc. Everything is defined at the module level.

_meta is only defined once we need a session. It's bad practice to give a lot of overhead to loading a module, particularly when there's a connection to an external resource involved. We only want to encounter a possible connection error once the programmer expects a connection to exist.

This particular session concept is from a wrapper in pykern. I thought about writing my own implementation, but the one Radiasoft provided is very convenient.

Copied from #20

nneveu

This is looking great! I want to look at the oracle and aida py files a little more, but nice work. I left some minor comments. Also, sorry to nag, maybe add a few comments.

kabanaty · 2026-03-11T17:28:14Z

+import slac_db.config
+import slac_db.oracle
+
+def to_oracle_db(csv_source=None):


Some docstrings would be helpful to indicate that this function is where the Oracle db is made, and defaults to lcls_elements.csv

Maybe name to_sqlite_db?

kabanaty · 2026-03-11T17:36:28Z

This is really great so far. I think docstrings and comments would be very helpful in interpreting the data flow and persistent session mechanics that are happening in this module.

shamin-slac

Looking good so far! I just had a couple minor comments, and I'll echo the request for more comments/docstrings.

shamin-slac · 2026-03-11T22:55:30Z

+    p = _Parser(csv_source=csv_source)
+    return slac_db.oracle.recreate(p)
+
+class _Parser():


Are the parentheses here to suggest that this takes an argument?

shamin-slac · 2026-03-11T22:58:23Z

+
+def recreate(parser):
+    assert not _meta
+    assert parser.addresses


Are these assert statements here for debugging? If not, I think it's better to not use assert statements for these kinds of checks.

Given the discussion in the workshop, should these be AssertError instead?

from slicops already

Eloise will add an error message

shamin-slac · 2026-03-11T23:43:42Z

+
+def recreate(parser):
+    assert not _meta
+    assert parser.rows


Same comment as earlier

nneveu

Yay, thanks!

nneveu · 2026-03-31T18:24:02Z

+import slac_db.config
+import slac_db.oracle
+
+def to_oracle_db(csv_source=None):


Maybe name to_sqlite_db?

nneveu · 2026-03-31T18:27:56Z

@@ -0,0 +1,15 @@
+import slac_db.aida
+
+def to_aida_db():


rename this to 'directory service'.

nneveu · 2026-03-31T18:36:01Z

+
+def recreate(parser):
+    assert not _meta
+    assert parser.addresses


from slicops already

nneveu · 2026-03-31T18:36:14Z

+
+def recreate(parser):
+    assert not _meta
+    assert parser.addresses


Eloise will add an error message

nneveu · 2026-03-31T18:38:36Z

rename file

…ql-interface-lfs

eloise-nebula added 5 commits March 5, 2026 14:51

Added a oracle/aida db creation from another repo.

16fd555

merge with main

290a28d

Added lfs tracking for sqlite3 databases.

670fc95

Added sqlite dbs

6e3a3cb

Tests pass and now we use LFS.

bac7f3a

eloise-nebula requested review from kabanaty, nneveu and shamin-slac March 9, 2026 23:46

eloise-nebula self-assigned this Mar 9, 2026

eloise-nebula added 2 commits March 10, 2026 09:14

Configure LFS in tests workflow checkout step

71e6793

Fix indentation in tests.yml for lfs option

80497e1

nneveu reviewed Mar 11, 2026

View reviewed changes

Comment thread slac_db/aida.py Outdated

Comment thread slac_db/aida.py Outdated

Comment thread tests/test_data/expected_areas.csv

Comment thread tests/test_data/expected_beampaths.csv

Comment thread slac_db/create/meme_names.py

kabanaty reviewed Mar 11, 2026

View reviewed changes

shamin-slac reviewed Mar 11, 2026

View reviewed changes

chkp

ba013eb

nneveu approved these changes Mar 31, 2026

View reviewed changes

eloise-nebula added 4 commits April 1, 2026 11:20

checkpoint

521530a

Checkpoint

6c81eeb

Merge branch 'sql-interface-lfs' of github.com:slaclab/slac-db into s…

0444c94

…ql-interface-lfs

Renamed to directory service.

8d008f2

eloise-nebula merged commit 3849dc6 into main Apr 1, 2026
6 checks passed

nneveu mentioned this pull request Apr 2, 2026

Develop SQL interface for AIDA and Oracle Tables #15

Closed

2 tasks

nneveu deleted the sql-interface-lfs branch April 9, 2026 23:45

	def _init_db(uri=None):
	global _meta
	if uri is None:
	uri = _oracle_uri()
	uri = _db_type_prefix(uri)
	schema = {
	"elements": {
	"Area": "str 64 nullable",
	"Element": "str 64 primary_key",
	"Control System Name": "str 64 nullable",
	"Keyword": "str 64 nullable",
	"Beampath": "str 64 nullable",
	"SumL (m)": "float 64 nullable",
	"Effective Length (m)": "float 64 nullable",
	"Rf Frequency (MHz)": "float 64 nullable"
	}
	}
	_meta = pykern.sql_db.Meta(
	uri=uri,
	schema=schema
	)

Conversation

eloise-nebula commented Mar 9, 2026

Uh oh!

nneveu left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

kabanaty commented Mar 11, 2026

Uh oh!

shamin-slac left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

nneveu left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants