v17.3.8March 08, 2017
🛠 We've switched to doing released in order to make sure folks are getting stable code. This release includes many changes (mostly bugfixes) from the past several months.
⚡️ Here is an abbreviated commit log. If you are updating your local sync engine, note there are several migrations you may need to run.
4df11df [sync] TravisCI testing changes and version bump for OSS release 5619ef9 [developer] More logs display bugfixes bcef868 [none] Update outdated comment 1b42c6c [admin] Add instructions for perf profiler and make easier to use 3e94a3f [balancer] Don't move Accounts that belong to debug sync instances d08e69b [api, sync] Handle more IMAP auth errors. Provide better error messages. aa9b82a [sync] Truncate long calendar names de16267 [api] Fixes AttributeError where some messages categories are None 5cc00df [sync] Truncate traceback to prevent DataError cont. dd7bad3 [sync] Truncate traceback to prevent DataError bcfa1a5 [sync] Make sure the thread exists 4f4bc6f [sync] Don't process event during message reconciliation 0cbd15f [sync] Fixes T7297 c386528 [*] Fix event queue bug 825d9bc [api] Don't schedule draft deletion until message is sent 29a5ed4 [all] Don't include debug instances by default 6123e2a [sync] Handle JSON parsing errors from event queue data 5a38ac6 [sync] Use KillerGreenletTracer for SyncHTTPFrontend 124b57d [sync] Use PPAs instead of CPU history for process limits 9de67a4 [sync] Unpack the right number of values for event queue lpop 2f17ab1 [migrator] Make migrator send heartbeats 3005676 [sync, api, admin, migrator] Guard event listener against multiple commits 622a37f [sync] Use correct end time for all day events b250782 [sync, api] Mark Accounts that already have post commit listeners 10243b5 [sync] Split shared and private event queue handlers 3b08cf4 [sync] Fix bugs in updated unschedule script 4b0b653 [balancer] Handle when account ID is missing 76f5b13 [sync] Update script to manually unload Accounts 4ed36b8 [balancer] Add stats logging for migration minimization d58d651 [sync] Flush private queue when receiving an event 2a80cd4 [balancer] Use Hungarian algo to minimize migrations 4b0040f [balancer] Parameterize the timespan for load balancing 2191519 [none] Delete legacy scheduler code 0a9626d [sync] Release owned Accounts that no longer should be 2daba2d [sync] Turn noisy event queue timeout statement down to DEBUG e850a9f [all] Make database transaction timing metrics optional 2a0719a [sync] Don't discard Account until it's marked as released in the DB 065eb63 [webhooks,sync,api,admin] Consolidate StatsD code, revise metrics names b950d0a [sync] Notify when Account.sync_should_run changes 8dc90fa [api] [direct_fetching] Handle case where g_msgid is None. 498a9ff [sync] Comment out new statsd counters for now 2c69eb5 [sync] Don't send migrate_from if desired_sync_host is None be41c73 [sync] Handle more folder roles edge cases fe8f3db [sync, api] Ignore irrelevent Account updates when sending events 9f65c7b Fix another draft deletion edge-case 1dcf4f9 [api] Add custom provider rule for adaptiveinsights.com 0f323fb [sync] Use gauge for sync_hosts_count metric 82b20fb [sync] Fixup sync restart script b8b1c27 [sync] Don't manually clear sync host ed70ec8 [sync] Add function to get vcpu count when balancing fleet 205fc95 [sync] Fix some problems with finding Account loads 1bfe66b [sync] Reduce some code duplication e67a9a5 [sync] Add script to dump per-Account loads in JSON 88d0184 [sync] Add scripts to dump and restore the assigned sync redis hash 531df0e [api, sync] Handle more errors during device provisioning. ad6ac16 [sync] Add legacy-balance-fleet to setup.py 1e2aa19 [sync] Add legacy load balancer 8b1ae86 [sync][api] Don't double-save to our legacy S3 bucket anymore aeae53a Log affected account id. 81e8407 [sync] Add script to restart sync for forgotten Accounts 5ed6047 [api] Direct fetching: handle non-existent UID replies. d8bb3e4 [syncback] Delete messages in a safe way 36812e6 [migrator] Moar load balancer fixes aea75f9 [migrator] More sync load balancer fixes 763fa83 [sync] Add script to balance sync fleet 226d322 [api] Try converting g_msgid to a number if we get a string. 3bd4764 [syncback] Merge adjacent SyncbackTasks for change_labels 2103eec [sync] Add service to schedule Account migrations across sync hosts e7eba03 [sync] Deduplicate migration events 8e015e2 [sync] Add per-Account sync counter for alerts b81bde9 [sync] Fixup string formatting in a couple scripts ca24daa Display exception breakdown in direct fetching validation script. d388df0 [sync] Add /build-metadata endpoint to SyncHTTPFrontend df7d351 More fixes for account migration c6c8a40 Fix Account migration stuff 7251aed [schema][sync, api] Add ability to migrate Accounts between sync hosts b76ad14 Fix SyncHTTPFrontend after refactoring ed580d6 Commit after undeleting a Thread c7d1da6 [api] Bump nylas-production-python version 0926449 [sync] Only collect first message metric for inbox c87a433 [sync] Include application_id in all requests 69df7aa [api] Add provider to api request logs a4cc16f [api] Log more details on direct fetch failures 8d56dc6 [api] Raise the right exception when trying to fetch a deleted message. 89c78bd Skip null MIME parts. 7aa87f4 Add more graphite logs. 49576e0 [migration, sync] Update imapfolderinfo uidnext to be a bigint instead of int 15b4d17 [direct-fetching] [sync] [api] Phase 2: Stop reading from our permanent bucket 480c75e [sync] Fix truncation for TEXT columns 95cc875 Add the ability to interrupt blocked greenlets 3b0cdce Fix alembic migration script revisions 4ed8cfc More exception catching 0da4345 Log attempts on individual syncback actions 66152e2 Standardize logging actionlog.id as action_log_id 0b77782 Log account_id and num_connections as structured data in crispin pool creation 4d468ee Make the direct fetching code a bit more robust e74efd2 Add profiler for SyncbackService e220f21 Add test for correct Contact.uid collation 881548b Revert "Remove Block/Part ForeignKeys" d7e2af3 Bump nylas-production-python version ab7bb85 Bump nylas-production-python version 9e4100b Skip Messages that don't have a valid Thread b8f8f81 [api, sync, schema] Make deleting Threads asynchronous 53fe179 [sync, api] Don't update remote events if the remote event creation failed f5b5d82 [api] Update fastmail provider settings 25e7a6f [api] Prevent TypeError for recurring calendar events 723d84c Fix incorrect rebase of alembic migration script 9781733 [SCHEMA] Drop foreign keys to and from Contact table e154aff Remove Block/Part ForeignKeys fe86721 [Ask @khamidou before shipping] Fetch attachments and raw messages directly from the email provider 3a81f0f [api] Don't leak Exception messages when we don't know what they are! 7ff64a4 Don't send an event invite to the owner. 5b99c86 Set Part.content_id for inline attachments 1f55c83 [api] fix(logging): Add all auth scope logging to WSGI context immediately 727f9dd [api] fix(logging): Log namespace IDs before throwing 404s e47e6e7 Add support for sending inline images 9a37aab [none] fix(logging): mysql “too many conn” is transient ed63006 [none] Upgrade coverage dep for more features 9028025 [sync] fix(logging): Get default logger if none provided in callback f308473 [none] fix(tests): Use a random port for SMTP server e4aa649 fix(logging): Clip error messages saved to Account objects 2d78db4 fix(logging): Don’t log transient mysql errors 3461444 Standardize test database names f388883 [sync] Bump IMAPClient to include folder_status fix 20f201f Provide detailed message to users when handling `OperationalError`'s 572b70b Roll back to older pytest 32e2b51 fix(tests): Unbreak local testing, oops 2f8b12e fix(tests): Skip test_block_on_depleted_pool() 7693683 Fix broken JSON in config-test.json 482203a Split test sync-engine shard config out into separate config files c88471a Remove redwood DB config from sync engine test config e3eb09b Disable webhooks tests during jenkins runs until they are fixed af0881f Add iontrading as an exchange provider 3512e7d Prevent ICS organizer CN field from being set to blank 20e121d Bump nylas-production-python version ab6d56a Don't allow DetachedInstanceError crashes in model repr() c873997 Fix passing in of echo to engine() b27868f Quiet noisy 'syncing events' log statement c48cb33 Fix from address. f5bf5a8 Fix typo which made code super slow. 5e13deb Turn down the production log volume more d7828d5 Remove Transaction ForeignKeys ea8d940 Give more insight about what deletion is doing 😅 485250a Fix dev secrets filename 113b95c Add ZONE in prod test config 4e9603e For prod too 236e968 Restore old SYNCBACK_ASSIGNMENTS values c860120 Syncback assignments really do have id '0' even though it's not valid JSON 435f06a Add back some keys to the sync-engine test config 7efb16b globs aren't strings 19be6ef Deduplicate test configs e9f80b7 Make 'syncing contacts' message DEBUG also 2f51947 Further reduce log volume 363f98d Remove extraneous conftest.py 87d8359 Install pytest.ini into package 49c4371 Pass 127.0.0.1 as the hostname to mysql when dropping/creating test tables f9d9b62 sync-engine system tests depend on unspecified nylas API client library 318e52e s/localhost/127.0.0.1/ on DATABASE_USERS also 7661f81 Use 127.0.0.1 instead of localhost in test mysql configs 1c02fee Give all test_util.py a different name e24c1d8 Fix SMTP SSL test cb1d356 Fix more tests ce226d9 Fix more tests in install da74be1 Ship JSON files too 511fb55 Fix a bunch of tests c6901b1 Fix up MANIFEST.in to ship test data 3abcc69 Fix Exchange tests to work in installed form c22eb3a Combine inbox.test dirs in installation 127e617 Make tests work when package is installed too 34276c6 Install test configs for sync-engine too 5928e13 Add tox to sync-engine deps 01fa0ac Distribute tests with source code
Hi Inbox Developers,
🚀 Over the past month we've been hard at work at Inbox HQ as well as around the globe to improve the Inbox Sync Engine, API, and SDKs, and are proud to announce the release of version 0.3.0. You can find all the gory details on GitHub, but the high-level changes are noted below.
🏗 While we strive to maintain API stability, we've been working with folks building apps on top of Inbox and have found that the following API changes make some apps drastically simpler—enough to justify a backwards-incompatible change.
- We've changed a number of endpoints to refer to items by
item_idsrather than just
itemfor greater clarity. This includes:
- Rename the
namespaceattribute of all objects to
- Rename the
threadattribute of messages and drafts to
- Rename the
draftsattributes of threads to
- Rename the
threadquery parameter for filtering messages and drafts to
- Rename the
- ⚡️ The Draft update endpoint now uses the PUT HTTP method, consistent with other objects like threads and messages.
- ⚡️ Drafts now have a
versionattribute, and updating a draft now simply changes its version, instead of creating a new draft object. (Note that updating or sending a draft now requires the
versionvalue as a parameter, in order to prevent write conflicts.)
🆕 New Calendar and Events APIs
📄 These new APIs allow access to calendar and event information for accounts from Google, iCloud, Hotmail/Outlook.com and Microsoft Exchange (contact [email protected] to request beta hosted access). Check out the docs to get started!
🆕 New SDKS
Here's a quick Python example:
inbox = APIClient(APP_ID, APP_SECRET, token) namespace = inbox.namespaces.first() for thread in namespace.threads.items(): print thread.subject
Other feature additions
- Attachment tag added to threads which have attachments.
- Create an
archivefolder where necessary for various providers.
- 👍 Namespace endpoint now support filtering options such as limit, offset and email_address.
- 👍 Files endpoint supports delete for files that are not currently attachments.
- 👉 User-created tags can be deleted.
🐛 Bug Fixes
- 📇 Propagate unread status changes to threads in update_metadata.
- 👍 Better handling of HTML stripping for snippets.
- ✂ Remove unused UIDNEXT from IMAP mailsync, since some providers don't support it.
- More graceful handling of various IMAP errors.
- 👌 Improved coverage for unit tests, notably over the API calls.
- ♻️ Refactor message / parts / blocks relationship to avoid creating new blocks when they are attached to messages.
- ✂ Remove duplicated code around SMTP mail sending.
- 📜 Truncate fields if they are too long on calendar/event parsing.
- Properly check to, cc, bcc fields in drafts for formatting.
With love from Team Inbox,
-- mg, spang, kavya, eben, ben, charles, karim, dwayne, and zoë
- We've changed a number of endpoints to refer to items by
Hi Inbox Developers,
🚀 We are pleased to announce a new release of Inbox. We are again excited to see the enthusiasm from the community and we've been hard at work integrating change requests, adding new features and fixing bugs.
🚀 Here's what's changed since the last release:
🆕 New Features
- 👌 Support for generic providers. We've now made it easier to add support for more providers through the "providers.py" file, where you may specify the relevant information for new providers without modifying other parts of the code.
- 👍 Basic multicore support for mailsync.
- 👍 Automatic detection for providers which support RFC4551 / 7162 (CONDSTORE).
- 👍 An improved threading algorithm for providers which do not support automatic threading on the backend.
inbox-authcommand is now scriptable with the options
- 👍 Better handling for invalid API query parameters.
- The API message representation now includes detailed attachment information such as content_type, size, and filename.
- 🎉 Initial support for Outlook.com/Hotmail including sending, receiving, archiving and draft management
- 👍 The drafts API supports
?thread=[filter], which returns all the drafts of a thread.
🐛 Bug Fixes
- 👌 Improved handling for accounts which no longer have valid credentials or cannot connect to the underlying provider.
- ⚡️ Updated html2text version to avoid issues with malformed html.
- 🔀 A significantly reduced number of database connections are now needed to sync accounts.
- 📇 Cleanly handle corrupted Gmail metadata cache files.
- SIGTERM is properly handled in
- The file attachment API call now returns a proper JSON response.
- Assorted drafts/sending bugs.
- 🛠 Many performance improvements and other various bugfixes.
- 👌 Improved provider identification during authorization process.
⬆️ If you're running Inbox on your own infrastructure, make sure you've disabled the sync, API and syncback processes and run the following to upgrade:
✅ Thanks to everyone for their continued support and interest in Inbox! If you run into problems with the latest, we hope you'll file an issue or hop onto ##inbox to get help.
🚀 The following folks contributed to this release (GitHub usernames by # of commits):
emfree, charlesg3, kav-ya, spang, khamidou, grinich, bengotow, mmattioli, rmasters