Integrating Neural Networks with LL(*) Parsers and Natural Language Processing

Natural language processing has been a traditional area of Artificial Intelligence. Transition Networks and Augmented transition networks have defined language grammars since 1970.

Java has taken grammars further with support for Java Speech Grammar Format (JSGF). The advantage of having JSGF is it makes it simple to parse back in results for more analysis.

Like AWK/SED parsers in Unix, JSGF takes the problem of stating a complex grammar further by allowing a straightforward syntax in text to represent a grammar. This opens the possibility of defining a self changing grammer.

The premier parser today is ANTLR4. It has versions for C and Java. But another possibility is the Apache OpenNLP which is much more primitive but provides access to source and some minimal trainability but is mostly a part of speech type old style NLP.

Then there are LL(*) grammars which are more useful in processing continuous speech. There are other techniques to help decide initial branch choices such as Probabilistic Recursive Transition Networks (the A now changed to R to demonstrate the more accurate issue of recursive definition)

But one issue is how to produce a grammer which is more like how Humans learn language. Could you define a infrastructure which itself is self learning and changing to pick up any language, whether German , Swahili, or abstract. That is quite a bit far off in the future.

But one area where neural networks can augment these systems is by providing learning and reinforcement training into weights and modifications of grammars and parsing trees.

One particular area which should have process are the ability to determine response especially to an interrogative. But also in the realm of general speech.

One problem with systems like SIRI and ALEXA is they are obviously rigid and stupid. Much of these systems have been done simply with collections of questions and statistics. Try to ask a reasoning question and they barf. Try to even ask them to convert your weather results to Farenheight and they barf. Why? Because they are not truly modeling language.

There are two primary issues involved. The first is it will require true semantic network modeling in order to provide more intelligent responses rather than a simple grammer with no knowledge representation. This is however a large effort which is going to require a revolution from the image based Neural Network paradigms like Tensor. This is why I invented the Neural Cube which is the next generation of Fukushima's Neo-Cognitron adding concepts from Karl Pribram and Gerald Edelman and John Koza.

Integrative systems will provide processing which can effect path choice weighting which can improve the efficiency of parsing grammars for real time speech or add deeper classifications to grammers - e.g. temporal location vs. noun.  They will also be able to tie in parallel neural networks to inject things like emotion, mode, task participation and state, and also tailoring response based on the relationship with the person. We don't want a cold robotic response like SIRI we want an empathetic response which shows understanding of the situation. This will become the principal feature when developing cybertrons for common human tasks such as working a hotel reception desk. People can be forgiving if the language or the process isn't perfect, as long as they form the empathetic connection. People become frustrated quickly if the effect is dealing with a cold machine, which as I hurl my Alexa device into the wall becomes ever so apparent.

Cybernetics and Cognitive Science stresses to build in adaptability and learning in many ways to make the systems work closer to how our brains work. While we have so much to learn in processing language, this is still an area where basic techniques can greatly improve the state of the art.

Noonean calls this "More natural language". Issues like emotive, goal, variability, and language that humans find pleasant is often more important than being able to answer complex questions correctly. That complex reasoning will come. But, the stoic rigid language of Siri and Alexa need to fade into history as a silly first attempt. It is still revolutionary, almost magic, but what is coming next will be language that doesn't annoy. Language that makes Humans comfortable. This is the promise of cognitive linguistics and cybernetic language development.

Quoted in a Thesis

I feel a bit like I've arrived when other researchers quote me in their work !

Bitter Electromagnet As Foundation Principle for Synthetic Muscles

The nylon twist coil heat activated synthetic muscle at first seemed promising. But ultimatly we could not get a reliable solution. They tended to break and fatigue, adding heat without melting the nylon was difficult. Braided carbon nanotubles was another solution but decades away in mass production as well as being too small.

So our synthetic muscle development is working on a principle which is used in Bitter electromagnets,which is a distributed helical coil field. This provides much of what we wish, a strong stable and efficient response to low voltage pulses.


The key aspect is how to manage contraction and retraction such that these can be durable for millions of pulses. But the unique Bitter design which is used in the worlds most powerful magnets, works just as well on the micro scale. As above so below ....

Using Tertiary Structures in HBM2 Arrays To Accelerate Neuronal Access

HBM2 architecture still suffers from bus latencies across the interposer substrate. Wider memory buses and stacked memory architecture still does not ameliorate the bus differential distance between the GPU processing core (stream processor) and HBM retrievals. This works find for mass bulk data screen loads supporting graphics but is a terrible architecture for neural processing.

Instead an interposed fine grained tertiary structure between HBM and each stream processor can improve performance by a factor of several thousand times for retrieval and update operations.

Nvidia's GP100 uses the same slow approach relying on high wattage bus speeds



The interposer tech is designed for one processor to access large scales of memory whereas the neuronal unit requires localized memory with hyperspeed interconnects which transcend each planar layer. A few random spiking units defer to interposing bus transactions alternating bit transfers and based on non-linear excitation strategies.

Here is a brief overview of correlet architecture for synaptic processing without large bus latency



This produces an inner group latency which is nearly instantaneous while larger outer group branches simply degrade exponentially according to distance. The main complexity with neural cube architecture is transposing non-linear synthetic distances into core relationships as its nearly impossible to dis-assimilate plane relationships which are inherently larger correlet hops than planar distance would indicate.



To ameliorate this situation, a plane dragging event is created to provide sequential ordered plane processing where required. Luckily in true convolution the cyclical stimulus is often non-ordered.

current implementations fail at this accelerated locality. Larger scale designs which operate on stimulation impedus rather than constant high rate bus cycling is the answer, something IBM has achieved with very small scale architecture (256 neural units) while true neural cubes begin to take form at 64 billion neuronal units. To get there synthetic networks transposing to high powered local groups is the optimal architecture. Eventual non-synthetic processing will be ideal, most likely when organic circuits develop.

Kurzweil the "father of AI" You've Got to be Kidding me!

A recent article

https://www.dailystar.co.uk/news/latest-news/682554/robots-artificial-intelligence-singularity-google-engineer-Jurgen-Schmidhuber-Ray-Kurzweil

calls out Ray Kurzweil as the father of AI. What HOGWASH.

First, I own a K2000RS and had a K1200. I know mr. Ray quite well. He is if anything the father of the speaking reading machine for the blind and sample instrument synthesis. NOT AI.

Let's talk the real history.

The Dartmouth Conference of 1956 was organized by Marvin Minsky, John McCarthy and two senior scientists: Claude Shannon and Nathan Rochester of IBM. ... At the conference Newell and Simon debuted the "Logic Theorist" and McCarthy persuaded the attendees to accept "Artificial Intelligence" as the name of the field

Marvin Minsky

Minsky is often the one I think of first when someone asks about founders of AI, and Turing pops to mind second simply because he did so much. 

Don't forget Kunihiko Fukishima's Cognitron who was so far ahead of his time its frightening, and a giant whose shoulders I stand on. 

Dr. Fukushima

 or Edelman's Neural Darwinism. Giants both. They forged a lot of principles of Noonean Inc. 

Dr. Gerald Edelman M.D.

What about Karl Pribram? Holographic Brain theory? 

Karl Pribram

What about the first commercial AI - Expert Systems. Nope Ray wasn't there either

developed by Edward Feigenbaum and his students. Dendral, begun in 1965, identified compounds from spectrometer readings. MYCIN, developed in 1972, diagnosed infectious blood diseases. They demonstrated the feasibility of the approach

Dr. Edward A. Feigenbaum

What about early neural networks? John Hopfield gave us those. His background was in molecular biology. Ray was mysteriously absent. 

John Hopfield

What about the founder of semantic nets - Ross Quillian? And what about companies? IBM and Bolt Beranack and Newman (sp?) which was whispered quietly as I was a senior at Vassar as how brilliant they were and how hard their interviews were. I ended up starting with IBM Research. But I wished I were smart enough for BBN. 

Ross Quillian

We ended up using Ross's work for enhancing search engines in 1996 and doing things miles ahead of the dummies at Google, but success isn't always about being the smartest. 

It seems the one big thing Kurzie did was write a rather poor book full of 30 year old research on AI, and then hire out of date hacks like Geoffry Hinton at Google. Lawd help us all.  Alright, Hinton is ok as a general theorist but he is hardly a pioneer and his Ted talk showed what I was studying in 1988!!!

Daniel Dennet led much of the work in symbolic representation in Cognitive science

Daniel Dennett

Dennett is our great american cognitive philosopher and still alive and going strong. May you live another 100 years Daniel we are in debt!!!

David Rumelhart was a brilliant psychologist and thinker into the development of mind. Sadly he is no longer with us. 

David Rumelhart

Roger Schrank a UT Austin alumn is still going strong as the CEO of Socratic Arts, a learning company. 

Roger Schrank

We also have Terry Winograd from Stanford's HCI group. He pioneered much of the work in natural language processing

Dr. Terry Winograd

Obviously an incomplete list. But I encourage to read and research these early pioneers and true fathers of AI. And hopefully as I am getting old and gray I will make the grade as well someday. These true fathers and founders, and 

As a aside, I am noted as the originator of the concept of the Neural Cube, a massive 3 dimensional self organizing structure of billions of neurons, as well as being a principal engineer in the early days of genetic programming. But I get few articles and I can't write my book without giving away too many secrets so.. that is on hold for now. But I will guarantee it will be light years past dim brain Kurzweils feeble mind. Is that harsh? Ray I love your chip wiring and early electronics, but a AI pioneer you were not. 

Book in Denmark...

I always marvel at how far and wide my books are sold... I really need to work with a translation house to get them available in native languages...

The Perils of Dependency Injection

When we worked with the Java Spring Framework (4.3 at the time) one of the things we had to adjust to was the fact that rather than have a J2E app server spring provided access to our realized objects through their dependency injection framework.

Now this is a lot like Windows versus linux. One time windows insisted on installing a security patch even though I had done everything I could possibly do to turn that off. So my computer rebooted twice, took several hours at load time with a threat "dont turn off!" and finally died altogether.

The idea of the computer taking control from you and believing "I know best" is really a new thing. One of the few pleasures we used to have was that computers unlike wives only did what we told them to do.

Now Spring is a lot like that. One of the first things that was confounding was what classes which used other classes which used other classes could actually be injected. This often wasn't as obvious as possible or predictable. Several times simple atomic classes which used other injects were not allowed. Why? This made no sense. It had to do with the internal class representation they had hacked into their injection representational model classes. It didn't have to make sense.

Another big issue which I think I mentioned before is how in the case of errors, again Spring would intercept the exception and re-wrote it to the callers callers callers class. The first time I tried to fix it with simple encapsulation. No dice it was thrown above that. Then we put in another set of classes specifically to get the throw. Nope still no dice. Finally we had a fully artifical layer calling a fascade calling into the classes. This made the code very hard to read and maintain. Why were we doing this? One developer suggested removing all this unnecessary code. Another did something similar the day of the big demo.

Now why WHY is this dependency injection stuff so high on the hog? I can always predict the classes I need from a constructor factory and have the same abilities. I've never had issues with getting hit with a new class type it just never comes up. Well a lot of it has to do with the MOCKIT or MOKITO type of testing. Testing can be difficult with the requirement to be in a fully live system. By using Mock object injection you can spoof your system to make it think its live and still test the function with the right state, say for example an object that has a handle which must be set by the system.

Again, yes useful, but real system tests always prove better. And rarely have I seen mockit object testing done well or ubiquitously it simply takes too long to setup.

Dependency injection seems like something they realized they could do, without actually looking at the pattern of use. does it have its place on some projects? Perhaps. But I advise caution.