Remove unused redis usage data

See merge request soprani.ca/sgx-bwmsgsv2!6

Simplify iq feature-not-implemented

See merge request soprani.ca/sgx-bwmsgsv2!5

This redis key is not used anymore

Send to the correct target, do not assume cheogram

See merge request soprani.ca/sgx-bwmsgsv2!4

Backport from prod hotfix

Log the errorCode and description for deliver-failure

See merge request soprani.ca/sgx-bwmsgsv2!3

Let them see the human readable failure description

deliveryCode and deliveryDescription were the v1 equivalents, but this is v2

Specifically https://gitlab.com/ossguy/sgx-catapult/commit/583f98b is
copied here.  While there was another commit in that merge request (at
https://gitlab.com/ossguy/sgx-catapult/merge_requests/20), that commit
(https://gitlab.com/ossguy/sgx-catapult/commit/8f9e588) was already
included in 9d556e1 so we don't need to copy the entire merge request
here, only the single commit in that merge request that included the
"Allow EMPromise.all to accept both Promise and Deferable" part.

Note that this is required in sgx-bwmsgsv2 since jmp-fwdcalls uses it
and jmp-fwdcalls must be able to run with either sgx-catapult or
sgx-bwmsgsv2.  As an example, the following commit and MR require it:
https://gitlab.com/ossguy/jmp-fwdcalls/commit/405ef3c and
https://gitlab.com/ossguy/jmp-fwdcalls/merge_requests/12 .

We are doing the copy here since we haven't properly refactored the
common parts of sgx-catapult and sgx-bwmsgsv2 so they can, for
example, share one em_promise.rb file.

See merge request !1 for the discussion and details behind the merge.

The metrics changes in .rubocop.yml definitely need to be reset to defaults, but
the required refactors are too big for this change alone so for now they were
increased to make it pass.

Other disabled rules may want to be revisited in the future also.

At least this passes its own ruleset now, and can run on sourcehut CI for every
commit.

This fix is ported in from sgx-catapult, where we made the fix in
https://gitlab.com/ossguy/sgx-catapult/commit/c8fd695 - it is fairly
simple, but also fairly important.  Here is the description (with
edits made to show the corresponding commit IDs in this repo):

Sometimes HTTP requests to the port that sgx-bwmsgsv2 is running on
are slightly less naive - for example, with a path of:

  /web-meetme/conf_cdr.php?bookId=1

In this case params is indeed non-empty so we need to also catch this
case.  To be extra safe, we do so by rejecting both non-POST and
non-"root" requests.  Hopefully this will suffice for the situations
we care about, though of course the better solution would be to do
actual request validation.  We'll save that for later.

As with eddafea we have the same error/crash without this fix (if we
got an HTTP request of the above form):

  Shutting down gateway due to exception 013: no implicit conversion of nil into String

A copy of https://gitlab.com/ossguy/sgx-catapult/commit/a887c61 :

This is not 100% correct, but it's close enough for our purposes and
we left a TODO in case someone wants to fix it properly in the future.
This fix will at least prevent feedback loops of the type we had
previously with components like Cheogram.

This fix is ported in from sgx-catapult, where we made the fix in
https://gitlab.com/ossguy/sgx-catapult/commit/300def9 - it is fairly
simple, but also fairly important.  Here is the description:

Without this fix, one will get the following error (and subsequent
crash) if the port that sgx-catapult is running on receives a naive
HTTP request, such as a "GET /" (from Wget or curl or similar):

  Shutting down gateway due to exception 013: no implicit conversion of nil into String

This is a cross-merge of the fix done to sgx-catapult in
https://gitlab.com/ossguy/sgx-catapult/commit/89598a8 - it has no
effect in sgx-bwmsgsv2 for now, but will help once we cross-merge the
blocking code (to be done later).

We just have one fix left until we otherwise have fix-/feature-parity
with sgx-catapult.

This is a cross-merge of what we did in sgx-catapult already, in
https://gitlab.com/ossguy/sgx-catapult/commit/f95aab3 - see
https://gitlab.com/ossguy/sgx-catapult/merge_requests/19 for further
details.  The merge is not exact, as there has been some drift in
sgx-bwmsgsv2 since this was done in sgx-catapult, so indentation and
some s/next/return/ caused it to be a bit off.  But it should be fine.

This effectively removes the option to turn off MMS-on-OBB, and since
that has already been removed in sgx-catapult (per above), we can now
remove it from jmp-acct_bot as there are no further users (to be done
in a future commit).

With this commit we are getting close to feature-/fix-parity with
sgx-catapult, aside from the message blocking feature (which we won't
need to merge for a while yet).

We run sgx-bwmsgsv2 and sgx-catapult off the same database, which
means that mere existence in the database does not mean that we can
pass on the message directly, as they could be on a different system
(i.e. sgx-catapult instead of sgx-bwmsgsv2) and Cheogram will become
unhappy if we try to pass on messages in this way (plus it's wrong).

As a result, we need to make the check more precise, and specifically
determine that the destination user is a user of sgx-bwmsgsv2, which
we do by checking for a distinctive characteristic of the credentials.
This should ideally be done in a better way, but since we just have
the two systems for now, it is easy to do this substring check, as all
credentials follow this general pattern.

We know this commit works because we've been running it in production
for a couple months now (* sigh *) and will be making a similar commit
to sgx-catapult in the near future (as it naturally requires the same
sort of change in order to work properly, per above).

While this localhost hardcoding would work fine if sgx-bwmsgsv2 was
run on its own, it usually isn't.  Instead, it is run as a plugover
(see https://gitlab.com/ossguy/jmp-fwdcalls/commit/08d3acc for
details), and because the voice endpoint that jmp-fwdcalls responds to
is public (so our carrier can deliver voice-related requests directly
to it), we need the whole endpoint to be public, even though the
requests delivered to sgx-bwmsgsv2 don't require that.

Ah, the idealism of 91ff289 could simply not be left unsullied.  We
though the V2 SGX would only need to deal with V2 media, but alas it
must deal with V1 media too, since we want it to work with the V1
jmp-fwdcalls.  In particular this means that send_media() must handle
the V1 media that is sent by jmp-fwdcalls (voicemail recording URLs).

This is roughly the counterpart of the fix we made at
https://gitlab.com/ossguy/sgx-catapult/commit/7dfe149683d31738076b075e9815643194d81684
- check the prefix and set the user accordingly (instead of checking
the user to set the prefix accordingly).

While we should probably document that deployment requires this extra
user, the warning in the code should serve the small number of people
who actually care well enough.  We hopefully won't need to run this
V1 + V2 monstrosity for very long.

With this fix we now fully support voicemails received by V2 users who
happen to still be using the V1 API for voice (which happens to be
everyone right now).  We thought this was all done as of
https://gitlab.com/ossguy/jmp-fwdcalls/commit/a4ed555e2dd16fb342f33ea7ee5675553baa39ec
but we needed the commit here to make voicemail recordings work.  I
suppose the jmp-fwdcalls commit message is still correct, though - no
additional work was needed in jmp-fwdcalls to fix the voicemail
recordings: only the fix here was required.

Since we ripped out support for Jingle File Transfer in 60832f5 we
need to figure out the best way to provide MMS by default to users of
this SGX (the V2 SGX).  Should we just turn it on by default for all
users?  We could, but we'd have to let anyone migrating from V1 to V2
know about this change.  And maybe there should be a different default
option?  In any case, it still needs to be decided so add a TODO here.

This is a continuation of the work started in 90a21bc, which prepares
the SGX for integration with jmp-fwdcalls and its V1 dependence.  Left
out from that previous commit was a mention of where we first did the
V1->V2 switch in this SGX.  For the URL code of important to the
call_catapult() method, this was in a9237ee.

Back to this commit, we have paved the way for jmp-fwdcalls to pass us
a config file containing the V1 credentials, which we can then use
when we detect a V1 path in call_catapult() and convert accordingly.

Until Bandwidth supports SIP endpoints in their V2 voice API, we need
to use their V1 voice API.  However, to use the V1 voice API alongside
the V2 messages API, we need to be a bit clever.  In particular, we
need to detect when we're getting a V1 call and adjust the URL
accordingly.

The hack implemented here to accomplish this allows us to make very
minimal changes to jmp-fwdcalls (letting it continue to use an
unmodified call_catapult() invocation) while still permitting it to
work with the V2 SGX (this one) as well as the V1 SGX.  As noted in
this commit's comments, jmp-fwdcalls will need to pass us some V1
credentials to use (since the V2 user won't have these available in
its creds list).  But that should be the only added complexity here.

We added group picture message receiving support in d89a1b7, but the
API introduced there (as part of the send_media method) was not ideal
due to the caller having to worry about the deep clone.  So instead we
will move the .copy() into send_media() so that it worries about the
deep clone and the caller doesn't have to care.

When we first added proper incoming group text support (in 90331bc),
we only supported 2 recipients.  We also assumed the first recipient
was the user, and then just truncated the rest of the recipient list
to be just one additional number, as a way of getting it implemented
quickly.

Now we've added a bit more thought, and a loop to go through all the
recipient numbers to add them to the address list we pass on to the
user.

We also needed to be careful to filter out our own number from the
recipient list, as that is already added in the addr1 section above.

Prior to this commit, picture messages sent to a group would appear to
come from just the sender, without any mention of the group (i.e. as
if the sender had sent the picture only to the SGX user).  With this
change, picture messages are correctly shown as being from the group,
utilizing the address list creation added in 90331bc.  Note that a
deep clone of the template message is required (via .copy) to ensure
that send_media() doesn't modify the original, which often contains
text that we need to send separately (done later in the code).

Since the media is stored at a completely different place with the V2
API, we need to update the send_media() method accordingly, so that
media (such as picture messages) is passed on correctly.

While this still technically requires an update to mpx-catapult, that
is beyond the scope of this project for now, particularly because it
is a very minor change there (it will remain in sgx-catapult for now).

So with this fix the URLs will be correct, assuming that the
mpx-catapult serving up this instance's URLs has been patched
accordingly.

We added some hacks to not send delivery receipts for group texts (as
they were not working correctly with Cheogram in our initial tests),
but we did not properly combine them when the second was added, in
part to keep the diff looking pretty.

So we'll do this now, noting that the first hack was added in d49bf12
and then a copy of it was added in a106c87.  Now they are combined.

This almost has no functional changes, except that if we were to
receive a message with an unknown type for a group text, the
corresponding message would no longer be printed due to this warning
and early exit occurring first.

This is similar to the support for message delivery receipts
themselves, so it's actually just as simple as checking for the
correct 'type', and the importing the two hacks that we already have
for message delivery receipts, namely those in b277759 and d49bf12 -
we probably want to move them into a common area eventually, but this
is fine for now, and definitely works in our tests just now.

While maybe a bit awkward (as the conditional is left until the end of
the statement), this fix for a non-present 'media' element sure does
make for an elegant diff.

With this, the SGX works at least as well as at it did before cd6e0df
(i.e. as of d49bf12) but the MMS ("picture messaging") support added
in cd6e0df does still need some work as mentioned there.  In
particular, we need to ensure that the media is accessible to the
user, which may necessitate us bringing back mpx-catapult in some form
(it was deleted in b1aabae).

Move the media handler code from the now-inactive 'mms' block (since
V2 messages don't have that type) into the new 'message-received'
block.  While this does cause the user to receive the media message,
there are still at least two problems:

1. This commit breaks non-MMS messages (since 'media' is undefined).

2. Media messages are not delivered with the correct URL (needs mpx?).

Anyway, we know about these, but wanted to keep this commit short and
sweet so that changes to the actual media handler are separate from
the commit that merely moves the code from one place to another
without changes to the media handler code itself.

Currently there is an interaction bug between this SGX and Cheogram
where Cheogram sends its "Instead of sending messages to cheogram.com
directly..." error if we naively send delivery receipts for group
texts.  This might be related to us sending two separate delivery
receipts with the same ID, but it's still unclear exactly what the
issue is.

So until we solve the problem that causes us to not be able to send
delivery receipts to Cheogram, just turn off that functionality.

This functionality wouldn't work anyway since Cheogram does not yet
support subscription requests on porcelain JIDs, which means we can't
test delivery receipts in most clients (as they need a subscription
before they will enable receipts).

Delivery receipts (at least for single-recipient messages) now work
correctly with the V2 API.  This was mostly simple changes (just
needed to change the deliveryState == 'delivered' check to instead be
type == 'message-delivered'), but did require a bit of a hack because
of the way we decided to determine the source and destination numbers
in d2e2d92 - there's probably a cleaner way, but we'll just go forward
with this quick fix for now, and do a better job of it later (which
may be needed anyway once we do (likely required) additional work for
multi-recipient message delivery receipts.

We discovered this while testing message delivery receipts, which
fortunately included the word "Gajim" in the messages in our tests,
causing Goliath to bail with an HTTP 400 error, indicating bad JSON.
Which was correct, since we would only have been sending the JSON part
before the 'G' in Gajim, due to incorrectly failing to limit the
number of splits being done by the split that is fixed here.

That will teach me to parse JSON without a JSON parser, or not...

Per https://wiki.soprani.ca/SGX/GroupMMS we have implemented the
protocol for receiving group texts.  For now we hard-code
"cheogram.com", but this and other items will be fixed in the near
future.

While the message going out does look correct, the current Cheogram
takes issue with it, sending us back the following error:

<message to="cheogram.com" type="error" from="<from>">
   <body>Instead of sending messages to cheogram.com directly,
   you can SMS your contacts by sending messages to
   +1&lt;phone-number&gt;@cheogram.com Jabber IDs.  Or, for support,
   come talk to us in xmpp:discuss@conference.soprani.ca?join</body>
...

So this code might not be complete, but we won't know for sure until
we're able to do a bit more Cheogram testing.  But it could be ready!

Also, we only currently handle group texts with 2 recipients - any
additional recipients will be dropped.  It should be trivial to add
more recipients by looping as needed.

Now we get a normal message via XMPP on a V2-style JSON blob, after
fixing the type info.  Group texts are not yet handled - that will be
next.

Now we get even further than in 6abd653 - since we have updated the
'to' so we use 'owner' instead, which is a string instead of a list.
In particular, we now get a message delivered (woo!), though it's of
the form:

'unknown type () with text: <text>'

So still a bit of work to do yet.  Also, we updated a couple other
fields with their appropriate analogs.

Thanks to 4607bb5 we now had params moved out of the way (since we
can't re-assign to this special variable) so we could now just simply
assign the JSON block to the new params variable (jparams).  Since the
Bandwidth V2 API differs a bit from the V1 API (on which this file is
based), we had to comment out some parameters for now, and change
'messageId' to 'id', but other than that the parameters at least
exist in V2.

However, this still doesn't cause the message to flow completely
through the SGX, since we haven't fixed the 'to' parameter, which is
now an array in V2, and so we now get something like:

'jid_key (catapult_jid-["+14165551212"]) DNE; BW API misconfigured?'

But this is way better than crashing the SGX, which is what we got
when passing in V2 messages before this fix.

So we're well on the road to getting V2 messages all the way through
the SGX and probably just need a couple minor fixes now.  Yay!

We previously sent the JSON blob that included the translator- and
acceptor-generated timestamps.  However, it is better is the SGX just
receives precisely the JSON that Bandwidth would have sent it, as that
makes it more easily compatible (i.e. if you want to use the SGX
without the acceptor and translator in front of it).  So we do that
here, being careful to pass on valid JSON (otherwise Goliath will have
an HTTP 400 fit).

Since it is easier to manage an HTTP connection between SGX and
translator, and this is likely how it will be used going forward,
remove the default of listening on all interfaces and instead just
listen on the loopback interface.  Managing HTTPS is beyond the scope
of the work here, and that would be required in order to safely listen
on other interfaces (and if you know it's safe for your situation, you
can of course just change the hardcoded value back to '0.0.0.0').

Updated a log message to remove TODO as we won't be putting the ID of
the message in the log, for privacy reasons.  This also means that we
won't have any need to read JSON anymore, so we can delete that
"require 'json'" line as well.

Added a TODO about handling another return value (of INCR).  This
probably isn't a big deal either way, as an INCR returning an error
multiple times will result in duplicate archived_message keys that
will be caught elsewhere, but there is still a small chance it won't
be caught, so it would be good to fix eventually.

We've taken care of a number of TODO items in this commit, including
checking if the archived_message key exists before setting it, adding
warnings about how this program should be treated as a singleton until
some extra atomicity is added, adding a couple TODO items, and finally
have the pending queue being managed correctly (checked at launch time
to confirm emptiness, then popped and checked again after processing a
message).

With this change, all of the queue state is now managed correctly, so
the translator can be run without needing to manually futz with the
queues between each invocation to get them in the right state.

However, there is still more work to do, as the translator does not
yet communicate correctly with the SGX (due to some of the remaining
TODO items, specifically related to how the HTTP request is sent).