OP, by "all in one place" do you mean one app (Photos) or one computer (Mac)? I'll suggest you consider the latter and organize actual photos in the Photos app and index the videos in the TV app. Both would be on a single Mac and could easily be synched to iDevices but they would obviously be managed in different apps.
I would also consider perhaps splitting the photo library into maybe a few Photos libraries, just to get them down in size. For example, it sounds like you are well organized based around time. So maybe split the libraries up like that. I don't know how many of each you have but it might be something like a pre-1990, 1991-2009, 2010-present.
Or maybe you split them all out by close family, more distant family (branches of the tree) and then friends & others.
Of course, you would choose whatever makes the most sense, but the core idea is to maybe split them out into a couple of libraries. Switching libraries is easy: hold down the option key when you open Photos and then choose the library you want to view/update.
The CATCH: if you want to sync photos from multiple libraries to iDevices, you can't do that (in the simple Finder way typical of synching). One library will be designated as main library and that's the one for synching. However, a good option there is to pick some "best of" classic photos you'd like to sync too and move copies of those into the main library. And of course, you can always transfer some other (alt libraries) photos to the main library at any time.
OR make a pass on all of the photos and create one new "best of" library with a subset of the big one (or ones if you take this advice) and let the "best of" library take over as main library for synching.
I have THREE Photos libraries: main, business, oldies. The oldies one is HUGE but it rarely gets any new updates. So it is stored as its own library, pretty much stays the same, and thus doesn't eat up lots of Time Machine (backup) space since it tends to be unchanged for long periods of time.
Business is simply business-related photos for clients and my own business, stock art, website art, etc.
Main, is the main library, which includes some "best of" photos from the oldies. In this one, I have many albums and I sync some of those albums to my iDevices.
This all works just fine.
A friend of mine likes using a phone like a document scanner (taking pictures of documents). So their library was pretty full of documents... which was often personal documents and she didn't really want them mixed together. So, we created a documents Photos library and deleted all of the photos from it (leaving only documents). Then, we deleted all documents from main library leaving only photos. This was ideal and a natural way to "split" a library. Private documents are in a library on the Mac. Photos are actual photos and can be more easily shared with other family members, viewed on AppleTV, etc.
On the video side, you can set up the TV app so that it indexing all of your videos but leaves them where they are (on the external drive) instead of importing them to the internal drive (and basically eating up very EXPENSIVE space- thanks Apple). If you are interested in this, let me know and I'll lay out the simple way how to do this.
Lastly, this kind of media tends to be the most personally valuable media on our computers. BE CERTAIN you have a great backup strategy that includes at least 2 full BACKUPS, with one of those always stored offsite. A classic & easy way to do this is use the free Time Machine app that comes with Macs and allocate 2 BIG HDDs for TM, letting it make full backups to BOTH drives. Store one offsite to protect against fire/flood/theft scenarios. Regularly rotate offsite with onsite HDD so that offsite is always pretty up to date. If you do this and suffer a catastrophe that takes out both the main Mac and the onsite TM drive, that offsite will restore the WHOLE archive for you. This is VERY IMPORTANT for this kind of media. Get on it if you do not already have a good strategy. Else, disaster is one iceberg away.
