Create meeting notes for sig-support #39 (#782)

* Create meeting notes for sig-support #39

Signed-off-by: George Appiah <[email protected]>

* sig-support Update 039-2024-11-20.md

added recording link

Signed-off-by: Drinkwater <[email protected]>

---------

Signed-off-by: George Appiah <[email protected]>
Signed-off-by: Drinkwater <[email protected]>
Co-authored-by: Drinkwater <[email protected]>

sig-support/meetings/039-2024-11-20.md

@@ -0,0 +1,175 @@
+# Akash Network - Support Special Interest Group (SIG) - Meeting #39
+
+## Agenda
+- Review of Issues Awaiting Triage
+- Updates on Bid Storm Mitigation and Provider Pods
+- Community Moderation and Hackathon Support
+- Vanguards Program Update for 2025
+- Open Floor for Feedback
+
+## Meeting Details
+- Date: November 20, 2024
+- Time: 07:00 GMT-7
+- [Recording](https://ne52qcczeqfriujvlfvwgehotnrdeyafkoiwfsoctoohul6n3f3q.arweave.net/aTuoCFkkCxRRNVlrYxDum2IyYAVTkWLJwpucei_N2Xc)
+- [Transcript](#transcript)
+
+## Participants
+- Tyler Wright
+- Scott Carruthers
+- Adam Wozney
+- Maxime Beauchamp
+- Damir Simpovic
+- Rodri R
+- Forbes Analos
+- B S
+
+## Meeting Notes
+
+### Opening Remarks
+- Tyler Wright welcomed participants to the monthly SIG Support meeting.
+- Emphasized the focus on triaging issues in the support repository and providing updates on community and technical support programs.
+- Acknowledged contributions from Maxime Beauchamp and the core product team in resolving console-related issues.
+
+### Review of Issues Awaiting Triage
+- Scott Carruthers reviewed two issues awaiting triage in the support repository:
+  - **Bid Storm Mitigation:** This issue arose due to increased bot activity targeting trial wallets. Existing mitigation measures include signed attributes for trial SDLs, ensuring only participating providers bid on trial workloads. Scott proposed adding further safeguards against future bid storms at the blockchain level.
+  - **Provider Pod Termination:** An issue where pods stuck in the "terminating" state after a node outage are not rescheduled. Scott outlined possible adjustments to provider builds to ensure forced termination and immediate rescheduling.
+
+### Updates on Bid Storm Mitigation and Provider Pods
+- Tyler Wright emphasized the importance of addressing bid storms and ensuring provider stability as key to maintaining network reliability.
+- Scott Carruthers committed to updating the issues with additional remediation details and testing outcomes.
+
+### Community Moderation and Hackathon Support
+- Adam Wozney announced upcoming changes to the Vanguards program for 2025:
+  - Vanguards will document recurring support issues monthly to inform the development of better documentation and support resources.
+  - Participation in SIG Support meetings will be added to their responsibilities, with flexibility for global time zones.
+- Rodri R highlighted ongoing support for the Akashathon:
+  - Noted increased activity and questions about hackathon submissions across Discord channels.
+  - Confirmed that technical queries are being directed to the dedicated hackathon support manager, PiberDev.
+
+### Vanguards Program Update for 2025
+- Adam Wozney discussed plans to reframe Vanguard responsibilities:
+  - Focus on documenting recurring support themes.
+  - Provide feedback through structured forms and meeting participation.
+  - Strengthen community support and technical onboarding.
+
+### Open Floor for Feedback
+- Participants shared feedback on current support practices:
+  - Rodri R noted improved spam management but recommended clearer moderation roles and the potential addition of a "Moderator" tag for better visibility.
+  - Adam Wozney acknowledged these concerns and committed to evaluating role clarity and anti-spam measures.
+
+### Wrap-up and Action Items
+- Tyler Wright thanked participants for their insights and contributions to the discussion.
+- Reiterated the importance of ongoing collaboration to address community concerns and improve support systems.
+
+## Action Items
+- **Scott Carruthers:** Update bid storm mitigation and provider pod issues with additional testing and remediation details.
+- **Tyler Wright:** Work with Adam Wozney to finalize Vanguard program updates for 2025, including structured documentation and meeting participation.
+- **Adam Wozney:** Evaluate and implement improvements to moderation practices, including role clarity and visibility for new users.
+- **Community Team:** Direct hackathon-related queries to PiberDev and monitor ongoing support themes.
+- **Maxime Beauchamp:** Continue addressing console-related issues and providing feedback to the support repository.
+
+
+## **Transcript**
+
+Tyler Wright: All right, welcome everybody to sync support. This is a monthly meeting now. It is November 20th, 2024. During the special interest group for members of the community and members of the core team go over any issues in the support repo that are awaiting  The support repo is where all issues that pertain to the Akash blockchain There is another the console repo for all things related to the Akash console. I know Max who's on this call here now and others from the core product team actively monitor issues, respond to people that want to help contribute etc. So always want to shout out Max and all the work they do.
+
+Tyler Wright: again the console repo there's issues in there where you can see if anybody wants to get involved there's usually good first issues and ways for people to get involved. outside of that again Scott from the core engineering team usually leads us through any issues that are awaiting triage inside of the support repo. So I will hand it over to Scott and we can get going.
+
+Scott Carruthers: All right, thanks I have a very minor cold. It's not really feeling bad, but it's just enough to affect my voice. So hopefully it's not too bad. let me share the screen. Is everyone able to see the issues  I'm seeing some thumbs up. so I sound great as always. Thanks. Thanks, W. My Darth Vader voice with a little bit of a cold. Sounds great. so I looked at this prior and there's just two issues that are awaiting triage. So we could probably go through these pretty quickly.
+
+Scott Carruthers: so the first one as you can see the one that was mo opened most recently was by one of our close community providers. and the title as you can see is bid storm mitigation. So this came up when trials were introduced on the Akash network and with the introduction of trials and trial wallets we saw a lot of bot activity. and when we have a provider that has a number of available GPUs, these bots might try to request a number of GPUs in a very short period of time. and just the trial activity in aggregate which producing a lot of bid activity. Max who's on the call can attest to this because we looked at this very closely with the console team to come up with some remediation.
+
+Scott Carruthers: So based on our current remediation, I think that we have mostly stemmied the immediate need and we did that by assigned attribute for console related trial SDLs. So there's now a signed attribute for that only providers that are participating in trials are signed with  So basically a provider which is a majority of the community providers on the network would no longer bid on trial SDLs. So we've largely mitigated this. the person that created this issue is not currently on the call. So I don't know if anyone else in the community Rodri or Max or any community members would have any perspective on this.
+
+Scott Carruthers: so I think at this point, and I'll probably do this after the call just so that I don't have to type it in real time, but as soon as we're done with the call, I'm going to put a comment into this that this issue has largely been remedied by the introduction of signed attributes. So again, community providers that are not participating in the trial will no longer bid. so there was a concern prior from the community members that bid activity had gone up to I think somewhere between $50 to $75 a day in provider fees just from this storm of bids that we had on the network. But again that's largely been remediated by steps taken.  So what I'm going to update this issue with is the details of what we've taken to this immediate concern. And then I think the question is do we still believe that there's a utility on the network for guarding against bid storms?
+
+Scott Carruthers: I don't think we have this concern again with trials any longer, but possibly to safeguard against other, new kinds of bid storms within the network. possibly we want to introduce some type of safeguard. So again, I'll detail this case just for our immediate purposes. I'm going to take this out of awaiting triage and I'll put it into the provider repo.  and I'll assign myself to it initially. so again, does you can just comment in the issue if you would rather and not or if you want to discuss it live here, that's fine as well. Does anybody have an opinion with the remediation that we've taken and now styate the immediate concern over trial bids?
+
+
+### 00:05:00
+
+Scott Carruthers: Does anybody have an opinion on if this issue still needs attention and we still need safeguards on bidtorm activity? But no one really has an opinion on that. on this caller live again I will update this issue immediately after and if anybody has an opinion I'll try to provoke comment from the original author to see if they believe there's cause to look at this issue any further.
+
+Scott Carruthers: 
+
+Tyler Wright: and I'll circulate it with some other folks that have talked about this issue in the past,…
+
+Tyler Wright: and I'll share with them and see if the issue still persists for them as well.
+
+Scott Carruthers: Okay.
+
+Scott Carruthers: There's obviously already ways to safeguard against this if your bid scripts only so this is very prevalent with most of our GPU providers on the network have safeguards in place so that they will bid on very low workloads. So if I create a Man SDL and have 0.1 CPU and very low memory and very low storage. So we're talking about extremely low cost workloads. there's already mechanisms in place that a provider could enact that they won't build on those workloads because they really only want to bid on GPU relevant and AI relevant workloads.
+
+Scott Carruthers: So there's already some safeguards against this but if we want to do it at a blockchain level we would have to investigate further. And with that this is actually from the same author which is DC North again one of our close community providers.  So I read through this issue before the call. and the concern here is if you have a pod running on a Kosh provider and for some reason that node were to go offline in maintenance activity or unexpected power outage just of a single node. So we're not talking about a clusterwide event but just a specific worker node goes offline. the concern here is that the pod will get stuck in terminating and won't be rescheduled on a different node.
+
+Scott Carruthers: so, there's possibly some ways in our provider build that we could make some adjustments so that defininity is not active and to ensure that as soon to force the termination of a pod when it goes into the terminating state and to reschedule immediately. so I'll take this out of awaiting triage and put it in the provider repo. Again, I'll assign this to Archer and myself.  Anyone have any opinions or thoughts on this issue? So, I'm just going to finish the assignment on this, but besides that, Tyler, I think we're all set.
+
+Scott Carruthers: 
+
+Scott Carruthers: So as I always offer, along with the issues that we just covered, if there's any other open issues that weren't awaiting triage, but, anyone in the community and…
+
+Tyler Wright: Go ahead, Adam.
+
+Scott Carruthers: on this call wants to discuss any other issues, feel free to speak up and we can cover any other issues that are on anyone's mind.
+
+Adam Wozney: yellow. So, not really an issue, but just something that I've been working on. So, as a part of kind of my 2025 community audit update governance proposal that I've been working on, specifically with the Vanguards, I've kind of reworked the requirements and some of the things that they are going to be needing to do going into 2025.  And one of those things is documenting support issues that they are dealing with across Discord or Telegram. so the idea is once a month at the end of the month they'll fill out a form that will have kind of the top themes of the month and then putting in any notes from specific individuals or things that they're noticing that are popping up. I thought this would be useful for many reasons. One is just to kind of document support issues that might not be noticed.
+
+
+### 00:10:00
+
+Adam Wozney: two is, taking some of this feedback and maybe creating documentation around, some of these support issues. and a few other things that I've been thinking about that are kind of relevant to this. So, I just wanted to kind of call that out. I'll probably reaching out to you, Scott, and maybe some of these other folks in this call to, see if this is something that, would be useful or making sure that this form that I'm creating would have the right information for folks to find useful. So, just wanted to call that out. Go ahead, Ty.
+
+Tyler Wright: I think that that's a great idea. part of the agenda for this SIG support call which is why Rodri BS and some people are oftentimes here is we get feedback from insiders and potentially vanguards in the community on things that they're seeing in Discord and other channels to see if there's any themes. I think writing those themes down would be great.  I would also love if possible for Vanguards to potentially be able to join this call sick support so it could be a way for them to connect with the community so we can get issues that are awaiting triage from the core engineering team and then anything else people can talk to Vanguards live we can start to talk about some of those themes and then have next steps where we can start to move them so I would love to see this
+
+Tyler Wright: 
+
+Tyler Wright: move forward. Again, we have a preliminary thought around that, but I think you're putting more structure around that. So, yeah, I would love to include again having Vanguards be more a part of this SIG support, which is a monthly meeting. Now, we have other meetings that are more frequent,…
+
+Tyler Wright: but if they can join this meeting as a part of their responsibilities, that would be Something to talk about.
+
+Adam Wozney: Yeah, that's actually No,…
+
+Adam Wozney: no, that's a great idea. I've been taking out I'm just kind of reframing a lot of the requirements and rewards for a lot of the different kind of community programs and that's actually a great idea to add this as a requirement. at a minimum watch the recording and leave feedback because they're around the world and this might be too late for some folks or whatever.  But making it a requirement I think is easy. so that's a great idea. I'm going to add that to the requirements.
+
+Tyler Wright: And we can always change this time. we try to do it this time because it works for most people across the world except for people on the West Coast, but we're always happy to adapt this time for you, Adam. Cool.
+
+Adam Wozney: Don't worry about it. I've been up since 2:00 a.m. This baby won't sleep.
+
+Tyler Wright: This is a great segue though because again after we triage any issues that are waiting triage in the support repo if there's nobody that has anything that they want to talk about in the support repo or the console repo I usually see if anybody here on the call I see a couple of folks that are very involved in the community here. if there's anything on Discord, Telegram, or in other channels themes or things that you've seen over the last month that might be worth calling out.
+
+Tyler Wright: I know Roger always gives his feedback on spam and some other things and we take some of those conversations to the insider office hours. But again, just want to see if there's anything that any insiders vanguards or just community members have seen in Discord, Telegram, or elsewhere over the last month that might be worth calling out.  Go ahead, Roger.
+
+Rodri R: Besides the usual spam and everything, which seems to be getting a little more or better every time. over the last past few weeks, there's quite a bit of attention over the hackathon. there's people asking about submitting and getting help with that several channels.  And I've also seen some people reporting that they've experienced some errors with predator app but tried to help and in the end I think it was PV and me and that replied to several of them and we sent them over to the parap channel because seems to be an issue with their RPC or something.
+
+Rodri R: Other than that, all good there. Haven't been much in Telegram.
+
+Adam Wozney: I can briefly talk to the hackathon support. So PyDED is our official technical community support manager for the Akashathon. So if you do see any hackathon questions that aren't getting responded to, just feel free to tag him and as soon as he's back online, he'll respond to those questions.
+
+
+### 00:15:00
+
+Rodri R: For sure.
+
+Tyler Wright: 
+
+Rodri R: Yeah. Sending people over to the hackathon support channel as well.
+
+Tyler Wright: Any other support call outs that anybody wants to make at this time?
+
+Tyler Wright: Is there anything else that folks want to talk about in terms of support? Again, there are some folks that have been joining other SIGs and working groups that have been talking about filing support tickets. Again, what we usually do is we usually have folks bring up issues that they're finding inside of Discord. Discord is where the builders are.
+
+Tyler Wright: And so there's a wide range of people in the community, insiders, vanguards as Adam has mentioned that are there almost 247 365 to provide again technical support if there's something that again can't seem to be solved by that group. There's an escalation path towards the core engineering team and folks can make issues that go in the support repo.  So again, if you see something and it's been escalated to that path and you've talked about it in another sacred working group, we ask that you make an issue inside the support repo so that we can talk about these things on a monthly basis during this call and look to get them rectified. People can talk about prioritization of issues as well.
+
+Tyler Wright: so again as always feel free to join this monthly call or get involved in side of the Discord channel. Cool. I know some folks are just joining us, but again we've talked about issues in the support repo. We've gotten some feedback on what's going on in Discord and other communities.  Adam has talked a little bit about the Vanguards program and some of the other programs around the community coming in 2025 or not even coming in 25 but continuing in 2025. just want to see if there's anything else that anybody want to talk about as it pertains to support before I let you all go. Most likely we will have another meeting next month because this is a monthly meeting and again there's always stuff going on.
+
+Tyler Wright: So we'll most likely have another meeting where if there's anything that's awaiting triage over these next couple of weeks. we'll again talk about at that time. Cool. If there's nothing else then I'll certainly let you all go. Again please stay engaged. There is a form on the website for those that's looking to get more involved and join the insiders. again, Adam talked about many updates coming to the community programs as we enter 2025. So look out for that recording and discussions there. Thank you to Scott as always for leading us through issues awaiting triage. Thank you to Max as always for addressing and being a part of the team that addresses any issues on the akash console side. appreciate everybody in the community as well for all your participation support of others. Hope everyone has a great rest of the day.
+
+Tyler Wright: There is a working group for the events 2025 later on today. I've just added to Discord. But again, appreciate everybody and we'll continue to talk online. Yes. Feel better, Scott. Bye-bye everybody.
+
+Scott Carruthers: Thanks. Bye, guys.
+
+
+### Meeting ended after 00:18:37 👋
+
+*This editable transcript was computer generated and might contain errors. People can also change the text after it was created.*