Completely migrate to prism

[tompng]

#1024
Builds on top of #1092

Remove remaining Ripper dependency and completely migrate to Prism.

RubyLex#should_continue?

Instead of checking Ripper's state bits, check for IGNORED_NEWLINE token.
When code is continued to the next line, there's probably a IGNORED_NEWLINE token.

1 + IGNORED_NEWLINE 2
receiver. IGNORED_NEWLINE method
receiver IGNORED_NEWLINE .method

RubyLex#check_termination_in_prev_line

When multiline code is pasted (without bracketed-paste mode), IRB checks if the last line is connected to the previous line.

tap do
end # Code looks ended here
.itself # But its' connected

Before: Heuristic, find DOT token
After: Compare two syntax trees: parse(lines.join) and parse(lines[...-1].join)

RubyLex#check_code_syntax

Checks if the code is :valid, :unrecoverable_error or :recoverable_error.
Same as before, use syntax error messages and regexp patterns.

Basic rules

• meets end of file end-of-input unterminated is likely to be recoverable error
• is not allowed unexpected or invalid is likely to be unrecoverable error
• invalid or unexpected error location is near last token position, it may not be unrecoverable error

Checked with many recoverable-error-code collected by

1. Collect valid short code snippets from ruby/ruby/**/*.rb
2. Randomly split code snippets into two, check if the first half is invalid code and join with "\n" becomes valid